TL;DR
This paper introduces the first taxonomy of in-document comment intents, classifying them into Modification, Information Exchange, and Social Communication. They also propose methods for automatically identifying these intents and explore how users interact with comments using voice commands, paving the way for more intelligent document collaboration tools.
What Problem Does This Paper Address?
Collaborative documents are rich communication channels, with comments acting as a dynamic way to discuss and refine content. However, managing these comments, especially in large documents, can be overwhelming. Current tools offer limited support for understanding the purpose behind a comment, making it difficult to automate comment handling. This paper addresses the need for a structured understanding of comment intents and explores how natural language interactions, specifically voice commands, can be used to manage them effectively.
Methods
The authors developed a novel taxonomy of comment intents by analyzing a large dataset of public documents. They then trained machine learning models, including classical methods like SVM and Logistic Regression, as well as advanced Transformer-based models (BERT, RoBERTa, BART), to classify comments based on their intent. A key aspect of their approach is the consideration of various contextual elements, such as the comment text itself, the selected text it refers to, and the surrounding paragraph.
To understand user interaction, they conducted a field study where participants used voice commands to manage comments within a custom-built web editor. This study involved collecting voice samples and telemetry data to analyze command patterns and user behavior.
Datasets
The paper introduces a new dataset of in-document comments. They collected approximately 1,000,000 .docx documents from the web, filtering them down to 1,313 documents containing comments. From these, they extracted 12,253 comments. A subset of 5,000 comments was manually annotated by three coders to establish the ground truth for intent classification.
Key Insights and Ideas
- Comment Intents are Diverse: Comments aren't just simple questions or suggestions; they serve distinct purposes like requesting modifications, exchanging information, or engaging in social communication.
- Context is Crucial for Intent: Understanding a comment's intent relies heavily on its surrounding text, not just the comment itself.
- Voice Commands for Comment Management are Feasible: Users can effectively interact with document comments using voice, and their commands often follow predictable patterns.
- Modification Comments Dominate: The dataset analysis revealed that modification-related comments are the most frequent, highlighting their importance in collaborative editing.
- Transformer Models Excel: Advanced models like BERT and its variants show strong performance in classifying comment intents, suggesting their potential for automation.
Key Results
- Taxonomy: The paper proposes a three-tiered taxonomy of comment intents: Modification, Information Exchange, and Social Communication, with further sub-intents.
- Classification Performance: Transformer-based models achieved high F1 scores (around 0.82) for classifying comments into the main intent categories, demonstrating the effectiveness of their approach.
- Voice Command Analysis: Voice commands for comment management are typically short (12-15 words) and often contain unique words not present in the document context. Users frequently use phrases related to comment actions (e.g., "delete the comment") rather than direct editing commands.
- Field Study Findings: Participants spent a median of 5-7 seconds on voice commands, with modification and information exchange commands being quicker than social communication ones.
Limitations and Future Work
The authors acknowledge several limitations:
- Crowd Worker Behavior: The field study used crowd workers who did not own the documents or know the other collaborators, which might lead to different interaction patterns compared to real teams.
- Custom Editor: Participants used a custom editor, which could have influenced their execution times due to unfamiliarity.
- Visual Impairment: The study did not include participants with visual impairments, a population that could significantly benefit from automated comment management.
Future work directions include:
- Conducting studies with real teams and in common text editors to validate findings.
- Exploring how people with visual impairments interact with document comments.
- Developing machine learning models for predictive comment management, such as predicting when a comment will be resolved.
- Further research into automating comment tasks beyond simple transcription and command execution.
Summary updated