A Conceptual Framework for Automatic and Dynamic Thesaurus Updating in Information Retrieval Systems

M.f. BruandetCOLING 1980

TL;DR

This paper proposes a novel methodology for automatically constructing and dynamically updating thesauri in information retrieval (IR) systems. By analyzing local document contexts and user feedback, it aims to create thesauri that better reflect the semantic content of a corpus and adapt to evolving information needs, moving beyond static, manually curated resources.

What Problem Does This Paper Address?

Traditional information retrieval systems often rely on thesauri to improve search recall and precision. However, manually constructing and maintaining these thesauri is a labor-intensive and time-consuming process. Furthermore, static thesauri struggle to adapt to the dynamic nature of information and the specific semantic nuances of a given document collection. This paper addresses the challenge of creating an automatic and dynamic thesaurus construction and updating mechanism that can learn from local document contexts and user interactions, thereby enhancing the relevance and adaptability of IR systems.

Methods

The core of the proposed methodology lies in analyzing local document contexts, defined as sets of documents retrieved by a specific query. A "metrical function" is introduced to quantify the distance between items (stems of words) within sentences. This function is then used to calculate a "binding strength" or association score, $b(s,t)$, between pairs of items $(s,t)$. This score is normalized to produce a "reference vector" $R_s$ for each item $s$, representing its related items and their association strengths.

The dynamic updating process involves comparing a newly computed reference vector $R'_s$ with existing reference vectors stored in a pseudo-thesaurus ($PS_x$) and the global thesaurus ($T_x$). Two functions, $m(s,t)$ and $M(s,t)$, are proposed to combine these vectors, effectively representing the union and intersection of related items. $m(s,t)$ considers items common to both new and old contexts, while $M(s,t)$ incorporates new associations. The paper outlines specific procedures for updating the thesaurus based on whether the new local context aligns with existing global or pseudo-thesaurus information.

For global thesaurus treatment, a similarity measure $r(x,y)$ is defined between items $x$ and $y$ based on their reference vectors in the global thesaurus. This measure can be used to cluster items into equivalence classes. The paper also touches upon feedback query processing, suggesting strategies to leverage both local and global thesaurus information to refine user queries.

Datasets

Not discussed in the paper. The methodology is presented as theoretical and oriented towards database implementation, with experimentation mentioned on a "small number of french texts."

Key Insights and Ideas

  • Local Context is Key: The paper emphasizes that understanding item relationships within specific local document sets (retrieved by a query) is crucial for building a relevant thesaurus.
  • Dynamic Adaptation: The proposed method allows the thesaurus to evolve over time by incorporating new associations discovered through ongoing queries and user interactions, moving away from static structures.
  • Bridging Local and Global: The methodology explicitly aims to integrate local semantic insights with global thesaurus structures, creating a more robust and adaptable system.
  • Fuzzy Set Theory Potential: The authors hint at the potential of fuzzy set theory for managing the complexity of global association matrices, suggesting a path for handling large vocabularies.

Key Results

The paper presents a conceptual framework and methodology rather than empirical results with quantitative metrics. The authors state that their experimentation on a small number of French texts suggests that the thesaurus updating method yields horizontal (non-hierarchical) relations. They also observe that unexpected associations reflecting the corpus's content emerge, and that "parasitic" items (those with weak or spurious associations) tend to decrease in strength over time, while synonyms (local and global) increase. The reliability of the thesaurus is noted to improve with a larger number of queries.

Limitations and Future Work

  • Theoretical Framework: The paper acknowledges that the implementation study has not yet been examined, and practical results are lacking.
  • Reliability and Stability: The authors note that the reliability of the thesaurus is only reached after a large number of queries. They suggest that new updating procedures might be needed to prevent the introduction of new "parasitic" items that could destabilize the thesaurus.
  • Future Goals: The authors outline two main future goals:
  • Making non-boolean queries possible through fuzzy keywords and improving dialogue.
  • Clustering documents using a dynamic indexing mechanism.
  • Global Thesaurus Scalability: The paper recognizes that building a global association matrix directly for a large number of items is not realistic and suggests exploring approaches like fuzzy set theory to overcome this.

Summary updated

BibTeX
@inproceedings{bruandet1980conceptual,
  author = {M.F. Bruandet},
  title = {{A Conceptual Framework for Automatic and Dynamic Thesaurus Updating in Information Retrieval Systems}},
  booktitle = {International Conference on Computational Linguistics},
  year = {1980},
}
Figure