TL;DR
This paper introduces a bilingual search system for Maharashtra government schemes that bridges the gap between informal citizen queries and formal administrative language. By fusing traditional lexical search (BM25) with modern semantic embeddings (MahaSBERT and IndicSBERT), the authors demonstrate that a hybrid approach achieves superior precision and ranking quality over either method alone, particularly in a bilingual English-Marathi context.
What Problem Does This Paper Address?
Navigating government services is often a nightmare of "vocabulary mismatch." A citizen might search for "financial help for farmers," while the official document is titled "Provision of Agricultural Credit." Furthermore, in states like Maharashtra, information may be authored in English but searched for in Marathi (or vice versa).
Standard search engines often struggle with these two hurdles:
- Semantic Gap: Exact keyword matching fails when users use paraphrases instead of formal terminology.
- Multilingual Barriers: Relevant documents might exist in a different language than the query.
The authors tackle these challenges by building and evaluating a retrieval pipeline specifically tuned for public service discovery in a bilingual environment.
Methods
The researchers developed a hybrid retrieval framework that operates in two parallel streams:
- Lexical Retrieval (BM25): Using Elasticsearch, they implemented a field-weighted BM25 algorithm. They assigned higher weights to titles and categories (3x and 2x respectively) compared to the main description to ensure that specific scheme names remain highly discriminative.
- Semantic Retrieval (Dense): They tested three configurations to generate vector representations of queries and documents:
- IndicSBERT: A multilingual model covering several Indian languages.
- MahaSBERT: A model specifically fine-tuned for Marathi.
- Translation-mediated MiniLM: A baseline that translates queries to English before using a standard English encoder.
- Score Fusion: The system combines the normalized scores from both streams using an equal-weight convex combination ($\alpha = 0.5$). This "hybrid" score determines the final ranking.
Datasets
The study utilizes a specialized, manually curated collection:
- The Corpus: 64 distinct Maharashtra government schemes and services, aligned in both English and Marathi.
- The Queries: 150 manually curated bilingual queries (75 English, 75 Marathi) mapped to relevant schemes.
- Alignment: English records were translated to Marathi using the Google Translation API, followed by domain-specific glossary checks and native-speaker reviews.
Key Insights and Ideas
- BM25 is a "Goliath": In specialized administrative domains, lexical matching is incredibly strong. Because scheme names and categories are unique and informative, BM25 alone achieved a high Mean Reciprocal Rank (MRR) of 0.864.
- Dense Retrieval is Not a Silver Bullet: On its own, semantic search performed significantly worse than BM25 (MRR ~0.66). This suggests that for government schemes, "meaning" alone isn't enough; the specific "keywords" of the law or department still matter immensely.
- Complementarity Wins: The "aha moment" is that while dense retrieval is weaker alone, it catches the paraphrases that BM25 misses. Fusing them provides a "best of both worlds" effect, pushing the MRR to 0.879.
- Language-Specific vs. Multilingual: Interestingly, the Marathi-focused MahaSBERT yielded the highest precision and MRR in the hybrid setup, while the multilingual IndicSBERT provided the best overall recall. This highlights a trade-off between depth (precision) and breadth (recall) when choosing encoders.
Key Results
The hybrid approach consistently outperformed the baselines across major metrics:
- Ranking Quality: The MahaSBERT-hybrid system achieved the highest MRR of 0.879, a modest but statistically relevant improvement over the 0.864 BM25 baseline.
- Precision: Hybrid fusion increased precision from 0.781 (BM25) to 0.808 (MahaSBERT-hybrid).
- Recall: The IndicSBERT-hybrid configuration reached the highest recall at 0.950.
- Relative Gain: While the jump from BM25 to Hybrid is modest (~1.7% MRR gain), the jump from Dense-only to Hybrid is massive (~34% relative improvement), proving that lexical signals are essential anchors for semantic models in this domain.
Limitations and Future Work
The authors identify several areas for growth:
- Scale: The current evaluation is limited to 64 schemes. Real-world government portals contain thousands, which may change the dynamics of "exact match" vs. "semantic" signals.
- Static Weights: The 50/50 split between lexical and semantic scores is arbitrary. Future work could explore query-dependent weights (e.g., giving more weight to BM25 for queries containing proper nouns).
- Eligibility: The system currently finds relevant schemes but does not determine if a user is eligible. Integrating structured eligibility logic with unstructured text retrieval remains a significant open challenge.
- Translation Nuance: Administrative Marathi is complex; the authors note that general-purpose sentence embeddings may still struggle with highly technical legal terminology.
Summary updated