Abstract: Mingxu Tao, Bowen Tang, Mingxuan Ma, Yining Zhang, Hourun Li, Feifan Wen, Ma Hao, Jia Yang. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (System Demonstrations).
2025.
Introduction
Navigating a university campus presents a unique "long-tail" information challenge. While general-purpose Large Language Models (LLMs) like GPT-4 excel at broad knowledge, they often lack the hyper-local, domain-specific data required for campus life—such as specific reimbursement policies, canteen operating hours, or the latest winter vacation schedules.
Traditional search engines frequently fall short in this environment; they are slow to index updated campus pages and perform poorly on cross-lingual queries (e.g., a French-speaking student inquiring about a Chinese-named building). Furthermore, campus data is inherently messy, existing in a mix of structured tables, FAQ lists, and unstructured web pages. ALOHA addresses these gaps by providing a multilingual "consulting agent" that is accurate, timely, and aware of information freshness.
System Architecture and Methodology
ALOHA utilizes a sophisticated three-stage pipeline designed to ensure precision and relevance: Preliminary Analysis, Hierarchical Retrieval, and Post-Processing.
- Preliminary Analysis: To manage multilingual inputs, the system uses FastText for language identification and the NLLB (No Language Left Behind) model to translate queries into Chinese. Chinese serves as the "pivot" language for retrieval since the vast majority of campus documentation is in Chinese. The system also employs Heuristic Intent Classification (HIC), which uses semantic similarity to categorize the user's goal into one of 11 specific domains, such as "Accommodation Expense" or "Opening Schedule."
- Hierarchical Retrieval: Rather than a flat search across all data, ALOHA prioritizes sources based on reliability. It first queries semi-structured documents (tables) for high-precision data. If no answer is found, it moves to unstructured documents, searching sequentially through "Concepts," then "QA Pairs," and finally "Web Pages." This "stop-early" approach prevents the model from being confused by noisy or conflicting information in lower-quality sources.
- Post-Processing: GPT-4o synthesizes the final answer. Crucially, it includes the timestamp of each piece of evidence in the prompt to ensure the model prioritizes the most recent information. Additionally, a Tool Planner can invoke external APIs to provide interactive features like live campus maps or canteen "Busy Indexes."
Datasets
The researchers developed a custom dataset to train and evaluate the system:
- Query-Intent Dataset: 717 seed questions were paraphrased using GPT-4o into a dataset of 3,573 query-intent pairs.
- Campus Document Store: A comprehensive collection of Peking University-specific documents, including handbooks, announcements, and news articles, categorized by structure and granularity.
- Human Evaluation Set: 100 real-world user queries covering academic and campus life, translated into English and French for multilingual testing.
Key Insights
- The "Pivot Language" Advantage: Translating queries to Chinese before retrieval overcomes the "cross-lingual gap" where search engines fail to find local results for foreign-language queries.
- Intent-Driven Filtering: HIC acts as a powerful filter, constraining the search space to avoid retrieving irrelevant but semantically similar documents.
- Hierarchical Retrieval as a Denoising Mechanism: By searching high-quality tiers (like tables and FAQs) first and stopping once evidence is found, the system drastically reduces hallucinations caused by conflicting information in general web crawls.
- Timeliness via Prompting: Providing the LLM with document timestamps allows it to reason about which information is outdated, a common pitfall for standard RAG systems.
Evaluation and Results
ALOHA demonstrated significant leads over commercial competitors in both accuracy and multilingual support.
- Accuracy: In Chinese queries, ALOHA achieved 94% accuracy, compared to 67% for ERNIE Search and 12% for Google Search.
- Multilingual Performance: ALOHA maintained high accuracy in English (87%) and French (78%), while competitors like ERNIE Search dropped to 0% for French due to a lack of cross-lingual retrieval capabilities.
- Intent Classification: The fine-tuned Qwen2-1.5B model, augmented with HIC, reached 96.3% accuracy in identifying user intent.
- Real-world Impact: Deployed as the "Xiaobei" assistant at Peking University, the system has served over 12,000 people and handled over 33,000 consultations.
Limitations and Future Work
While highly effective, the authors noted that machine translation can occasionally fail on specific entity names, leading to retrieval errors. Future work will focus on better entity alignment across languages to ensure terms like "Canteen" map perfectly to their Chinese proper names. The authors hope ALOHA can serve as a blueprint for other public affairs and educational agents globally.
Summary updated