Natural Language Query Refinement for Problem Resolution from Crowd-Sourced Semi-Structured Data

Rashmi Gangadharaiah, Balakrishnan NarayanaswamyIJCNLP 2013 · OpenReview

Abstract

Rashmi Gangadharaiah, Balakrishnan Narayanaswamy. Proceedings of the Sixth International Joint Conference on Natural Language Processing.

2013.

Introduction

Technical support agents often face the challenge of resolving customer issues based on "underspecified" initial descriptions. A user might report that "Bluetooth isn't working," omitting critical context like the specific device or a recent software update. While online product forums contain vast amounts of troubleshooting data, this information is unstructured and noisy. Traditional decision support systems often require extensive manual labeling or historical query logs, which are frequently unavailable for new or proprietary products.

This paper introduces an interactive Decision Support System (DSS) that mines crowd-sourced knowledge from online forums to automatically generate natural language follow-up questions, helping agents narrow down problem "signatures" and reach solutions more efficiently.

System Architecture

The proposed system operates in two distinct phases: an offline knowledge extraction phase and an online interactive refinement phase.

Overview of the proposed two-phase Decision Support System.

1. Offline Phase: Knowledge Extraction

The system crawls forum threads (e.g., Apple Support Communities) and extracts three types of Representational Units from the initial posts:

  • Phrases: Common noun phrases such as "interactive web sites."
  • Attribute-Value Pairs: Modifiers identified through dependency parsing, such as "signal: strong."
  • Action-Attribute Tuples: These are 4-tuples representing (Subject-Verb-Object-Context), such as "I-entered-dns-null." This novel representation captures the specific actions users take that define a problem.

Extracted units are scored using tf-idf to filter out conversational filler and focus on informative content.

2. Online Phase: Interactive Refinement

When a user submits a query, the system retrieves initial results and performs:

  • Unit Selection: Using Information Gain (IG), the system identifies the unit that most effectively discriminates between the retrieved candidate solutions.
  • Question Generation: The selected unit is mapped to a natural language template. To ensure the question sounds natural, the system utilizes linguistic features like tense and person from the original forum text (e.g., "Has the update caused the phone to crash?").

Key Insights: Robustness through Syntax

A core challenge in processing forum data is the variety of ways users describe the same event. Simple keyword matching often fails to bridge these gaps. The authors leverage dependency parse trees to identify underlying semantic relationships, making the system robust to variations in phrasing.

Dependency parse trees for variations of "sync server failed" showing consistent semantic relationships.

By extracting "Action-Attribute" tuples, the system can recognize that "server failed," "server has failed," and "server has been failing" all point to the same core issue. User studies confirmed the value of this approach: 37% of the most helpful follow-up questions originated from these tuples. Furthermore, by treating query refinement as an Information Gain problem, the system minimizes the number of interactions required to reach a solution.

Evaluation and Results

The system was evaluated using approximately 147,000 threads from the Apple Discussion Forum (2007–2011). A test set of 217 queries across 60 common problem areas was used to simulate real-world behavior.

  • Performance: The baseline search achieved a Mean Reciprocal Rank (MRR) of 0.3997. The full system, incorporating Action-Attribute tuples, significantly improved this to 0.7362.
  • Interaction Success: The probability of finding the correct solution in the top 5 results increased markedly with each follow-up question.
  • Human Evaluation: In a study of 100 generated questions, 87% were rated as understandable. Failures were primarily attributed to errors in the underlying dependency parser when handling informal text.

Conclusion and Future Work

While the system demonstrates significant improvements in problem resolution, it remains sensitive to parsing noise in informal text. Future work aims to incorporate "social" signals, such as poster reputation or solution upvotes, to improve ranking. Additionally, the authors plan to explore more sophisticated natural language generation techniques beyond template-based approaches.

Summary updated

BibTeX
@inproceedings{gangadharaiah2013natural,
  author = {Rashmi Gangadharaiah and Balakrishnan Narayanaswamy},
  title = {{Natural Language Query Refinement for Problem Resolution from Crowd-Sourced Semi-Structured Data}},
  booktitle = {IJCNLP},
  year = {2013},
  url = {https://openreview.net/forum?id=dLAAv_H4hyn},
}
Figure