TL;DR
This paper proposes using condensed text representations, specifically key phrases and key sentences, generated by an extended TextRank algorithm, to speed up document annotation without sacrificing accuracy. A user study found that annotators could label documents using key phrases twice as fast as full-text documents, with comparable accuracy, suggesting a significant cost reduction for training data generation.
What Problem Does This Paper Address?
Manually annotating documents for text classification is a crucial but time-consuming and tedious task, especially when dealing with large datasets. The quality and quantity of this training data directly impact the performance of classification models. While existing methods like active learning aim to reduce the amount of data needed, this paper focuses on reducing the time it takes to label each individual document. The challenge is to find representations that are concise enough to speed up annotation but still retain enough information for accurate classification.
Methods
The core of this work lies in an extension of the TextRank algorithm, a graph-based ranking algorithm originally designed for keyword and key sentence extraction. The authors propose TextSentenceRank, which unifies both key phrase and key sentence extraction. The key innovation is a graph construction where words and sentences mutually reinforce each other's importance. For key phrase extraction, they use a specialized tag cloud layout that reflects the relationships between phrases without imposing a strict linear order, acknowledging the potential loss of sequential information.
The evaluation methodology involved a user study where participants annotated documents under three conditions:
- F (Full-text): Standard document view.
- S (Key Sentences): Documents represented by extracted key sentences.
- P (Key Phrases): Documents represented by extracted key phrases and named entities, displayed as a tag cloud.
Datasets
The authors used a German news corpus from the Austrian Press Agency, consisting of 27,570 news articles from 2008. This corpus was fully labeled with five classes: "economy," "sports," "culture," "politics," and "science." For the evaluation, they selected articles longer than the 3rd quantile (> 337 words) and excluded statistical outliers (> 655 words), resulting in 6,328 articles. The key phrase extraction evaluation also utilized a subset of 500 documents from the Hulth (2003) dataset, which was also used in the original TextRank paper.
Key Insights and Ideas
- Condensed representations can significantly speed up annotation: The study strongly supports the idea that presenting users with summarized versions of documents (key phrases or sentences) drastically reduces the time required for labeling.
- Key phrases offer a sweet spot for speed and accuracy: While key sentences also speed up annotation, their accuracy is significantly lower. Key phrases, on the other hand, provide a good balance, allowing for fast and accurate labeling.
- User perception doesn't always match measured performance: Participants perceived themselves as most accurate with full-text documents, yet measurements showed comparable accuracy with key phrases. Conversely, they felt most stressed and least informed by key phrases, despite performing fastest. This suggests a need to build user trust in these condensed formats.
- Classifier performance is robust to minor mislabeling: The study found that the slight decrease in labeling accuracy when using condensed representations did not significantly impact classifier performance, at least with the tested classifiers and dataset size. This implies that the cost savings from faster annotation might outweigh the minor accuracy drop.
Key Results
- Speed: Users labeled documents using key phrases twice as fast as full-text documents (10.4s vs. 19.9s on average). Key sentences were also faster (10.7s vs. 19.9s).
- Accuracy:
- Key phrases and full-text conditions showed comparable accuracy in terms of correctly labeled documents (7.84/10 and 8.24/10 respectively).
- Key sentences had significantly lower accuracy (7.59/10).
- Classifier Accuracy: The difference in classifier accuracy when trained on original labels versus user-labeled data was less than 4% across all conditions, and these differences were not statistically significant.
- User Preference: Participants perceived full-text as most helpful and least difficult, while they found key phrases most stressful and least informative, despite performing fastest.
Limitations and Future Work
- Generalizability: The authors acknowledge that further investigations are necessary for different classification tasks and languages.
- User Trust: The discrepancy between perceived and measured performance, particularly the lack of user trust in key phrases despite their accuracy, needs to be addressed. This might involve user training or more familiar interfaces.
- Classifier Robustness: While current classifiers were robust to minor mislabeling, the impact might increase with larger datasets and more complex classification problems.
- Future Directions: The authors suggest exploring other languages, hierarchical/multi-label classification, varying keyword extraction parameters, and conducting eye-tracking studies to understand reading patterns and inform adaptive user interfaces. They also propose combining condensed representations with full-text or using highlighted keywords.
Summary updated