A Chatbot for Asylum-Seeking Migrants in Europe

Bettina Fazzinga, Elena Palmieri, Margherita Vestoso, Luca Bolognini, Andrea Galassi, Filippo Furfaro, Paolo TorroniarXiv 2024 · DOI · arXiv

TL;DR

ACME is a neuro-symbolic chatbot designed to help migrants navigate the labyrinthine European asylum system by identifying the highest level of protection they may qualify for. By combining Large Language Models (LLMs) for natural language understanding with a symbolic "computational argumentation" engine, the system provides transparent, auditable legal guidance that significantly outperforms general-purpose models like GPT-4o.

What Problem Does This Paper Address?

The European asylum process is notoriously complex, involving multiple tiers of protection (e.g., Refugee Status, Subsidiary Protection, and Special/Humanitarian Protection), each with distinct legal requirements. Migrants often lack the legal literacy to know which status to apply for, leading to a "shotgun approach" that overburdens territorial commissions and courts.

While LLMs like ChatGPT are fluent, they are ill-suited for this domain due to three critical flaws:

  1. Hallucinations: They may invent legal rules or precedents.
  2. Lack of Transparency: Their reasoning is a "black box," making them unauditable for legal applications.
  3. Reasoning Gaps: They struggle with complex, multi-hop reasoning across multiple legal documents and specific, non-public jurisprudence.

Methods

The authors propose a neuro-symbolic architecture that decouples natural language interaction from legal reasoning.

  • The Knowledge Base (KB): Instead of a flat database, the KB is an Abstract Argumentation Framework. It consists of "status arguments" (factual conditions like "victim of human trafficking") and "reply arguments" (legal outcomes). These nodes are connected by "attack" (contradiction) or "support" (endorsement) relations.
  • Natural Language Module: This module maps user input to KB nodes using a three-tiered approach: direct string matching, semantic similarity via SentenceTransformers, and a fallback to an LLM (Llama 2 70B) to infer concepts not explicitly stated.
  • Argumentation Module: This is the symbolic "brain." It uses a dialogue strategy that doesn't just look for a matching reply; it seeks a consistent and defended reply. If a user's facts support a status but that status is "attacked" by other potential facts, the system proactively asks follow-up questions to resolve the conflict.
  • Privacy by Design: Because the reasoning engine only sees the mapped KB nodes, any irrelevant personal information shared by the user is discarded, ensuring data governance.
Fig. 2: An excerpt of our Knowledge Base

Datasets

The system was validated using a specialized case study focused on international protection in Italy for migrants from Nigeria.

  • Expert-Curated KB: Developed with immigration lawyers, the KB includes 26 argumentative nodes covering gender, country of origin, and specific vulnerabilities (e.g., victims of rituals, economic precariousness).
  • Synthetic Profiles: The authors generated 10 simulated migrant profiles with randomized factual backgrounds to test the system's accuracy against human experts.

Key Insights and Ideas

  • Robustness Over Fluency: ACME distinguishes between "consistent" and "potentially consistent" replies. It will delay giving an answer if the current facts could still be defeated by future information, prioritizing legal accuracy over immediate gratification.
  • Auditability: Because the system is symbolic, it can generate "why" and "why not" explanations by tracing the paths of endorsement and attack within the argument graph.
  • LLM as a Component, Not the Core: The paper demonstrates that LLMs are best used as NLU "translators" to map human speech to formal logic, rather than as the primary reasoning engine for high-stakes legal tasks.
Fig. 1: Overview of the interface.

Key Results

The authors conducted a double-blind experiment comparing ACME against both a human legal expert and GPT-4o:

  • Human-Level Accuracy: ACME agreed with the legal expert on 10/10 cases, correctly identifying the highest level of protection for every simulated profile.
  • LLM Failure: GPT-4o (zero-shot) only identified the correct protection 3/10 times.
  • Lack of Robustness: GPT-4o gave different answers to the same case when the prompt was slightly altered. Alarmingly, the authors noted that the LLM's incorrect justifications "sounded like expert advice," highlighting the danger of using raw LLMs for legal guidance.

Limitations and Future Work

ACME is currently a proof-of-concept prototype with several acknowledged limitations:

  • Narrow Scope: The current KB is limited to Nigerian migrants in the Italian legal context.
  • No Credibility Assessment: The tool assumes the user is telling the truth. In real court cases, a judge's primary task is often assessing the credibility of the applicant's story, which ACME does not attempt.
  • Future Directions: The team plans to expand the KB to cover more nationalities, integrate speech-to-text for better accessibility, and add automatic translation modules to support migrants who do not speak the host country's language.

Summary updated

Abstract

We present ACME: A Chatbot for asylum-seeking Migrants in Europe. ACME relies on computational argumentation and aims to help migrants identify the highest level of protection they can apply for. This would contribute to a more sustainable migration by reducing the load on territorial commissions, Courts, and humanitarian organizations supporting asylum applicants. We describe the background context, system architecture, underlying technologies, and a case study used to validate the tool with domain experts.
BibTeX
@misc{fazzinga2024chatbot,
  author = {Bettina Fazzinga and Elena Palmieri and Margherita Vestoso and Luca Bolognini and Andrea Galassi and Filippo Furfaro and Paolo Torroni},
  title = {{A Chatbot for Asylum-Seeking Migrants in Europe}},
  year = {2024},
  eprint = {2407.09197},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.09197},
}
Figure