Turiya at PerpectiveArg2024: A Multilingual Argument Retriever and Reranker

Sougata Saha, Rohini SrihariACL 2024

TL;DR

This paper introduces "Turiya," a novel multilingual argument retrieval system designed to account for diverse socio-cultural contexts. By combining a sophisticated retriever with an ensemble of LLM and MLP-based re-rankers, Turiya significantly outperforms existing methods in retrieving relevant arguments across different languages and cultural perspectives, addressing a critical gap in current argumentation technologies.

What Problem Does This Paper Address?

While argument retrieval systems have become quite adept at finding relevant arguments based on semantic content, they often fall short when dealing with the nuances of human communication across different cultures and languages. The "Perspective Argument Retrieval" (PAR) task highlights this challenge: simply finding semantically similar arguments isn't enough. We need systems that can understand and factor in socio-cultural aspects like persona, attitude, and demographics, which can profoundly influence how arguments are perceived and whether they are considered relevant. This is particularly crucial for making argumentation technologies more inclusive and accessible to a global audience. The paper tackles this by aiming to retrieve arguments that are not only semantically relevant but also align with the socio-cultural background specified in a query or associated with the arguments themselves.

Methods

Turiya employs a two-stage architecture: a retriever followed by a re-ranker.

Corpus Processing: * Arguments from a multilingual corpus (German, French, Italian) are first translated into English using Mistral-7B-Instruct-v0.2 in a zero-shot setting. * English embeddings are generated using multi-qa-mpnet-base-dot-v1, and multilingual embeddings are created using paraphrase-multilingual-mpnet-base-v2. * An Hsnwlib (graph-based vector index) is populated with English embeddings for efficient approximate K-nearest neighbor (KNN) search.

Retriever: * The retriever generates two query embeddings: one English translation of the multilingual query and one multilingual embedding of the original query. * It then performs two types of retrieval: * Cosine Similarity: Computes cosine similarity between the query's multilingual embedding and the corpus's multilingual embeddings to get a ranked list. * KNN Search: Performs KNN search on the Hsnwlib index using the query's English embedding to retrieve another ranked list. * The final retrieval ranking is an average of these two rankings, empirically determined to yield the best results.

Re-ranker: * The re-ranker is an ensemble of two models: * MLP Re-ranker: A 2-layered neural network that takes 768-dimensional query and document embeddings, encodes them, concatenates them, and outputs a relevance score. It's trained on a custom dataset (rel) achieving an F1 score of 73% on the dev split. * LLM Re-ranker: Fine-tuned Mistral-7B-Instruct-v0.2 model trained on custom instruction datasets (rel and mcq) using LoRA for parameter efficiency. This model achieves an F1 score of 79% on the rel dataset and 51% accuracy on the mcq dataset dev splits. * During inference, the re-ranker enlists the top 30 documents from the retriever, prioritizing those deemed relevant by both the MLP and LLM models.

Datasets

The paper utilizes the following datasets:

  • Argument Corpus: Comprises 26,335 arguments from the 2019 Swiss Federal elections in German, French, and Italian, enriched with eight socio-cultural properties and covering 45 political aspects.
  • Queries: Based on the x-stance dataset, covering 35 political aspects for training and 10 for development.
  • Test Sets: Three secret test sets for final evaluation.
  • Custom Datasets (rel and mcq): Constructed internally for training the re-ranker.
  • The rel dataset (6,150 examples: 4,815 train, 1,335 dev) consists of query-document pairs with binary relevance labels, using English-translated text.
  • The mcq dataset (8,795 examples: 6,895 train, 1,900 dev) involves tasks where a query and two documents are presented, requiring the identification of the more relevant one or if they are equally relevant/irrelevant. This dataset was created to increase task variety for the LLM re-ranker.

Key Insights and Ideas

  • The Power of Ensemble Re-ranking: Combining a traditional MLP-based model with a powerful LLM like Mistral significantly boosts re-ranking performance. This suggests that different model architectures capture complementary aspects of relevance.
  • Multilingual Embeddings are Crucial: The system leverages both English-translated and original multilingual embeddings for retrieval. This dual approach allows it to capture semantic meaning effectively while also retaining language-specific nuances.
  • Socio-cultural Context Matters: The PAR task and the explicit scenario evaluation demonstrate that explicitly incorporating socio-cultural information can lead to more relevant argument retrieval, moving beyond purely semantic matching.
  • Custom Datasets Drive Performance: The creation of rel and mcq datasets was essential for training the re-ranker models, highlighting the need for tailored data to address specific challenges like perspective-aware retrieval.

Key Results

  • Baseline Scenario: Turiya significantly outperforms BM-25 and Sbert baselines on test sets 2 and 3, and achieves comparable results to Sbert on test set 1, demonstrating robustness across diverse test sets. For instance, on test set 2, Turiya achieves an NDCG score of 0.94, compared to Sbert's 0.88 and BM-25's 0.78.
  • Explicit Scenario: Turiya significantly outperforms the Sbert baseline across all test sets, achieving NDCG scores ranging from 0.60 to 0.72, compared to Sbert's scores between 0.13 and 0.44. This highlights the system's effectiveness in handling explicit socio-cultural matching.

Limitations and Future Work

  • Computational Cost: While not explicitly detailed as a limitation, the use of LLMs for re-ranking inherently implies higher computational costs compared to simpler retrieval methods.
  • Implicit Socio-cultural Understanding: The paper focuses on explicit socio-cultural matching in the "explicit" scenario. Future work could explore how to better infer and utilize implicit socio-cultural cues from queries and documents.
  • Broader Language and Cultural Coverage: The current system is trained on Swiss-German, French, and Italian. Expanding to a wider range of languages and cultural contexts would be a natural next step.
  • Deeper LLM Integration: Further exploration of LLM capabilities for both retrieval and re-ranking, potentially in a more end-to-end fashion, could yield even better results.

Summary updated

BibTeX
@inproceedings{saha2024turiya,
  author = {Sougata Saha and Rohini Srihari},
  title = {{Turiya at PerpectiveArg2024: A Multilingual Argument Retriever and Reranker}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure