ALOHA: Empowering Multilingual Agent for University Orientation with Hierarchical Retrieval

Mingxu Tao, Bowen Tang, Mingxuan Ma, Yining Zhang, Hourun Li, Feifan Wen, Hao Ma, Jia Yang

Abstract

The rise of Large Language Models (LLMs) revolutionizes information retrieval, allowing users to obtain required answers through complex instructions within conversations. However, publicly available services remain inadequate in addressing the needs of faculty and students to search campus-specific information. It is primarily due to the LLM's lack of domain-specific knowledge and the limitation of search engines in supporting multilingual and timely scenarios. To tackle these challenges, we introduce ALOHA, a multilingual agent enhanced by hierarchical retrieval for university orientation. We also integrate external APIs into the front-end interface to provide interactive service. The human evaluation and case study show our proposed system has strong capabilities to yield correct, timely, and user-friendly responses to the queries in multiple languages, surpassing commercial chatbots and search engines. The system has been deployed and has provided service for more than 12,000 people.

Introduction

Navigating a university campus—especially for freshmen—is a daunting task involving fragmented information sources: student handbooks, frequently updated department websites, complex reimbursement tables, and building schedules.

General Large Language Models (LLMs) often "hallucinate" when asked about these niche domains because they lack specific campus data. Meanwhile, traditional search engines struggle with "timeliness" (often surfacing outdated pages from previous years) and "multilingualism" (failing to find local Chinese campus info when queried in English or French). ALOHA addresses these gaps by building a system that understands user intent, retrieves the right type of document, and integrates real-time tools like campus maps.

System Architecture and Methods

The ALOHA system architecture, featuring a hierarchical retrieval-augmented generation (RAG) pipeline.

The ALOHA system operates in three distinct stages designed to maximize retrieval accuracy and user experience:

  1. Preliminary Analysis:
    • Language Identification: Uses FastText to detect the user's language.
  2. Translation: Translates non-Chinese queries into Chinese to maximize retrieval success, as most campus documents are in Chinese.
  3. Intent Classification: Uses a finetuned Qwen2-1.5B model combined with a Heuristic Intent Classification (HIC) method. HIC uses semantic similarity to narrow down the possible categories of a query, which significantly boosts accuracy.
  4. Hierarchical Retrieval: Instead of searching all documents at once, ALOHA categorizes data by structure and quality. Semi-Structured data (tables like schedules or financial standards) are prioritized if the intent classifier detects a need for precise data. If no tabular data fits, the system searches sequentially through Concepts (definitions), then QA Pairs, and finally Web Pages. It stops as soon as high-quality evidence is found to reduce noise.

  5. Post-Processing:

    • Generation: GPT-4o generates a response grounded in the retrieved evidence, explicitly prompted with document timestamps to ensure timeliness.
  6. Tool Integration: If a user asks for a location, the system acts as a "Tool Planner," providing hyperlinks to interactive campus maps or "busy indexes" for canteens.

Experimental Setup

Datasets

  • Document Store: A vast collection of Peking University-specific documents, including handbooks, news articles, and crawled web pages.
  • Intent Dataset: 3,573 query-intent pairs generated by taking 717 "seed" questions and using GPT-4o to paraphrase them into various styles.
  • Evaluation Set: 100 real-world user queries covering academic development, campus life, and career planning, translated into English and French for multilingual testing.

Key Insights

  • Specialized Small Models: A tiny, finetuned 1.5B parameter model (Qwen2) outperformed the massive ERNIE-4.0 at intent classification (96.3% vs 92.3%), proving that specialized components can be more effective than general-purpose giants.
  • Intent-Driven Retrieval: By identifying whether a user needs a specific rule (from a table) or a general explanation (from a web page) before retrieving, the system avoids the noise common in flat RAG systems.

Results and Impact

Performance comparison of ALOHA against commercial chatbots and search engines across multiple languages.

ALOHA was compared against ChatGPT (GPT-4o), ERNIE Search, and Google Search. The results demonstrate significant improvements across all key metrics:

  • Accuracy: In Chinese queries, ALOHA achieved 94% accuracy, compared to ERNIE Search (67%) and ChatGPT (29%).
  • Multilingual Performance: For French queries, ALOHA maintained 78% accuracy, while ERNIE Search dropped to 0% (as it refused to serve non-Chinese queries) and Google Search sat at a meager 1%.
  • Timeliness: ALOHA correctly identified 2025 winter vacation dates, whereas Google Search surfaced outdated 2024 information.
  • Real-world Impact: The system (deployed as "Xiaobei") has already served over 12,000 people and handled 33,000 consultations.

Limitations and Future Work

While ALOHA represents a major step forward, challenges remain. Errors in the initial translation step (e.g., confusing technical acronyms) can propagate through the pipeline. Additionally, the system relies on periodic crawling; future work will focus on more robust automated extraction if website formats change. Finally, the team aims to implement more robust "fact-checking" steps to further mitigate the risk of LLM hallucinations.

Summary updated

Abstract

The rise of Large Language Models~(LLMs) revolutionizes information retrieval, allowing users to obtain required answers through complex instructions within conversations. However, publicly available services remain inadequate in addressing the needs of faculty and students to search campus-specific information. It is primarily due to the LLM's lack of domain-specific knowledge and the limitation of search engines in supporting multilingual and timely scenarios. To tackle these challenges, we introduce ALOHA, a multilingual agent enhanced by hierarchical retrieval for university orientation. We also integrate external APIs into the front-end interface to provide interactive service. The human evaluation and case study show our proposed system has strong capabilities to yield correct, timely, and user-friendly responses to the queries in multiple languages, surpassing commercial chatbots and search engines. The system has been deployed and has provided service for more than 12,000 people.
BibTeX
@misc{tao2025aloha,
  author = {Mingxu Tao and Bowen Tang and Mingxuan Ma and Yining Zhang and Hourun Li and Feifan Wen and Hao Ma and Jia Yang},
  title = {{ALOHA: Empowering Multilingual Agent for University Orientation with Hierarchical Retrieval}},
  year = {2025},
  eprint = {2505.08130},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.08130},
}
Figure