Recovering Problem-Solving Activities from Query Messages
Abstract
This paper presents a computational framework for reconstructing the underlying logic of a user's troubleshooting process from their natural language help requests. By mapping specific Japanese linguistic patterns to a structured "Problem-Solving Graph" (PS-graph), the system can infer a user's goals, failed actions, and the causal relationships between them, even when the user's own domain knowledge is flawed. Evaluation on a corpus of technical support messages demonstrates high precision in identifying problem-solving structures, offering a path toward more intelligent automated support systems.
Introduction
When users post to technical support forums, they rarely provide a clean statement of their goals. Instead, they describe a complex journey involving failed attempts, unexpected side effects, and shifting objectives. For an automated support system to be truly helpful, it must first understand this "problem-solving activity."
The primary challenge lies in the ambiguity and incompleteness of human descriptions. A user might state, "I tried X and then Y happened," leaving the system to determine if Y was a goal, a side effect, or a new problem. This research aims to bridge the gap between raw text and a formal representation of the user's mental model, specifically within the context of troubleshooting narratives.
Methods
The authors propose a three-stage "Recovery Mechanism" to transform raw text into a structured graph.
1. Fragment Identification
The system first scans the message for "surface expression patterns"—specific linguistic markers that signal a problem-solving step. In Japanese, certain modal expressions are powerful indicators of a user's mental state. For instance, the verb suffix -te-shimau (often used for the perfect tense) frequently signals that the speaker views the result as an undesirable "problem," even without explicit labeling.
2. Sub-graph Assignment
Each identified fragment is assigned a "ready-made" sub-graph based on its problem type. These sub-graphs utilize an ontology of four node types: G (Goal), P (Problem), X (Action), and XP (Composite).
3. Integration
The system incrementally links these sub-graphs into a comprehensive PS-graph. To manage the potential combinatorial explosion of possible links, the authors introduce two key heuristics based on typical human behavior:
- Principle of Alternative Actions: A user typically only tries a second-best option if the first one fails.
- Principle of Decomposed Actions: A user will not proceed to the next step in a sequence if the current step fails.
The system also handles "virtual actions" ($X?$). When a user describes a goal and a sub-action but skips intermediate steps, the system automatically inserts these virtual nodes to maintain logical integrity.
Datasets and Evaluation
The study utilized a corpus of messages from the Usenet newsgroup "fj.sys.mac" (a forum for Japanese-speaking Macintosh users) from 1993. The analysis focused on 42 troubleshooting messages, from which 63 distinct problem description fragments were extracted to test the mapping rules.
The mapping rules were evaluated via hand simulation, yielding an average precision of 90.2% and an average recall of 73.0%. The system proved highly accurate at correctly identifying problem types when a prediction was made.
Discussion and Limitations
A key insight of this work is its ability to handle "wrong" knowledge. Traditional plan recognition often assumes a correct "plan library." This framework acknowledges that users often act on incorrect beliefs. By focusing on the structure of the activity rather than domain-correctness, the system can recover the user's logic even when their actions are technically nonsensical.
However, the study has limitations. The evaluation was performed on a relatively small sample and relied on hand simulation rather than a fully automated implementation. Furthermore, while the linguistic patterns are general to Japanese, the specific corpus is tied to 1990s computing. Future work involves implementing the mechanism in a real-world system to evaluate performance on larger, modern datasets and further formalizing troubleshooting as a set of mental processes.
Summary updated