Abstract
Online forums contain valuable human-generated information. End-users looking for information would like to find only those threads in forums where relevant information is present. Due to the distinctive characteristics of forum pages from generic web pages, special techniques are required to organize and search for information in these forums. Threads and pages in forums are different from other webpages in their hyperlinking patterns. Forum posts also have associated social and non-textual metadata. In this paper, we propose a model for online thread retrieval based on inference networks that utilizes the structural properties of forum threads. We also investigate the effects of incorporating various relevance indicators in our model. We empirically show the effectiveness of our proposed model using real-world data.
Introduction
Generic web search engines typically operate under the assumption that a single webpage constitutes a single document. However, in the context of online forums, this assumption is often invalid. A single discussion thread can span multiple pages; returning a middle page without the context of the initial question or the surrounding debate significantly degrades the user experience. Furthermore, forum data is characterized by "noise" links—such as login pages and user profiles—that can disrupt traditional ranking algorithms like PageRank. Effective forum retrieval requires a model that respects the unique hierarchical structure of threads and leverages the social signals inherent in community interactions.
Methodology: Inference Networks for Thread Retrieval
To address these challenges, we employ Inference Networks, a type of Bayesian Network, to model the retrieval process. This approach allows for the integration of multiple sources of evidence within a unified framework.
Structural Weighting
Rather than treating a thread as a monolithic text block, our model decomposes it into three primary structural components: the Title (T), the Initial Post (I), and the Reply Posts (R). Each component is assigned a specific weight ($\alpha$), enabling the system to learn the relative importance of keywords appearing in different parts of the thread. For instance, a keyword in a title may carry different weight than one found in a late-stage reply.
Incorporating Social and Structural Priors
Beyond textual content, we incorporate query-independent "priors" that serve as indicators of thread quality:
- Thread Length: Longer threads often suggest more comprehensive or popular discussions.
- User Authority: We distinguish between "Information Seekers" and "Information Providers." Posts from established providers are weighted more heavily.
- Link Information: We construct a "Forum Graph" where links between threads are weighted based on the authority of the user who created the link.
Datasets and Evaluation
The model was evaluated using two large-scale datasets: the Ubuntu Forums (a technical support community with ~113k threads) and TripAdvisor (a travel discussion community with ~83k threads). Relevance judgments were provided by human annotators for 25 queries per dataset to establish a gold-standard evaluation set.
Results and Discussion
Our experiments demonstrate that structural modeling significantly outperforms baseline "whole thread" retrieval. On the TripAdvisor dataset, the combined structural model (T+I+R) achieved a Mean Reciprocal Rank (MRR) of 0.7351, a substantial improvement over the 0.6445 baseline.
Key insights from our evaluation include:
- Reply Dominance: Reply posts (R) were found to be the strongest individual indicators of relevance, as they typically contain the actual solutions users seek.
- Title Limitations: Searching by title (T) alone yielded the poorest results, likely due to the prevalence of non-descriptive titles (e.g., "Help me!!") in forum posts.
- Social Signal Impact: Incorporating Link Information (Li) provided the most significant performance boost among the priors, pushing MRR to 0.7567 for Ubuntu and 0.8067 for TripAdvisor.
Conclusion
By adopting an inference network approach that accounts for the structural and social characteristics of online forums, we can significantly improve the accuracy of thread retrieval. Future work will address challenges such as topic drift in long threads and the complex interactions between different types of metadata priors.
Summary updated