Mots composés dans les modèles de langue pour la recherche d'information

Carmen Alvarez, Philippe Langlais, Jian-Yun NieTALN (Posters) 2004 · OpenReview

Abstract

Carmen Alvarez, Philippe Langlais, Jian-Yun Nie. Actes de la 11ème conférence sur le Traitement Automatique des Langues Naturelles. Posters.

2004.

Introduction

Traditional Information Retrieval (IR) systems frequently rely on the "bag of words" assumption, treating documents and queries as independent collections of unigrams. While computationally efficient, this approach fails to capture the semantic context provided by word combinations. Previous attempts to incorporate structural information using bigrams (adjacent word sequences) have often proven too restrictive for IR tasks. For instance, a query for "apartment rentals" should ideally match a document containing the phrase "rent an apartment," yet a strict bigram model would treat these as distinct events due to word order and intervening terms.

This paper proposes a language modeling approach that utilizes Lexical Affinities (LAs)—flexible, unordered word pairs—to capture semantic relationships more effectively than rigid bigrams while maintaining the robustness of unigram models.

Methodology

The authors introduce a new language model, $M_{LA}$, which integrates relational information through Lexical Affinities.

Defining Lexical Affinities

An LA is defined as an unordered pair of words $<u, v>$ appearing within a fixed window (typically five words) of each other. To maintain a consistent representation, pairs are stored alphabetically. The importance of an LA is determined by its resolving power ($\rho_d$), a metric analogous to tf-idf. A pair is considered highly characteristic of a document if it appears frequently within that specific document but rarely across the broader corpus.

Extracted table from page 4 of Mots composés dans les modèles de langue pour la recherche d'information

The $M_{LA}$ Model and Smoothing

Rather than expanding the vocabulary with millions of distinct bigram-like terms, the $M_{LA}$ model incorporates LAs into a unigram framework via "fractional counts." The document model is enriched by adding counts based on the resolving power of the LAs it contains. To address the sparsity problem (zero-probability for missing query words), the authors employ linear interpolation, smoothing the document-specific model with a global corpus model.

Experimental Setup

The model was evaluated using the TREC AP90 collection, comprising 78,321 documents from the Associated Press. Testing involved 53 queries from TREC-6 and TREC-7, categorized into two formats:

  • TITRE (Title): Short, keyword-based queries (average 2.5 words).
  • DESC (Description): Longer, natural language descriptions (average 7 words).

Results and Discussion

The $M_{LA}$ model was compared against the SMART vector space model and standard unigram/bigram language models using Mean Average Precision (MAP).

Performance comparison (MAP) of the $M_{LA}$ model across various parameters against SMART, 1-gram, and 2-gram baselines.

Key Findings

  • Superiority over Baselines: The $M_{LA}$ model consistently outperformed the SMART system and the standard unigram model. For short "TITRE" queries, it achieved a relative gain of 2.0% over the unigram baseline. For longer "DESC" queries, the gain was approximately 1.0%.
  • Flexibility vs. Rigidity: $M_{LA}$ outperformed standard bigram (2-gram) models. This confirms that the strict adjacency requirement of bigrams is a hindrance in IR, whereas the proximity-based approach of LAs captures underlying concepts more robustly.
  • Short Query Impact: The most significant improvements were observed in short queries. In these cases, capturing the relationship between the few provided keywords is essential for disambiguation and precision.

Conclusion

By moving beyond the bag-of-words model and incorporating flexible Lexical Affinities, the authors demonstrate a measurable boost in IR performance. The $M_{LA}$ model effectively bridges the gap between simple unigrams and overly restrictive bigrams. Future work may involve refining LA selection through part-of-speech (POS) filtering to reduce noise and focusing on noun-adjective pairs to further enhance semantic precision.

Summary updated

Abstract

Carmen Alvarez, Philippe Langlais, Jian-Yun Nie. Actes de la 11ème conférence sur le Traitement Automatique des Langues Naturelles. Posters. 2004.
BibTeX
@inproceedings{alvarez2004mots,
  author = {Carmen Alvarez and Philippe Langlais and Jian-Yun Nie},
  title = {{Mots composés dans les modèles de langue pour la recherche d'information}},
  booktitle = {TALN (Posters)},
  year = {2004},
  url = {https://openreview.net/forum?id=cjdIV_wcvK8},
}
Figure