Abstract
With the rapid accumulation of text data produced by data-driven techniques, the task of extracting "data annotations"—concise, high-quality data summaries from unstructured raw text—has become increasingly important. The recent advances in weak supervision and crowd-sourcing techniques provide promising solutions to efficiently create annotations (labels) for large-scale technical text data. However, such annotations may fail in practice because of the change in annotation requirements, application scenarios, and modeling goals, where label validation and relabeling by domain experts are required. To approach this issue, we present LabelVizier, a human-in-the-loop workflow that incorporates domain knowledge and user-specific requirements to reveal actionable insights into annotation flaws, then produce better-quality labels for large-scale multi-label datasets. We implement our workflow as an interactive notebook to facilitate flexible error profiling, in-depth annotation validation for three error types, and efficient annotation relabeling on different data scales. We evaluated the efficiency and generalizability of our workflow with two use cases and four expert reviews. The results indicate that LabelVizier is applicable in various application scenarios and assist domain experts with different knowledge backgrounds to efficiently improve technical text annotation quality.
Introduction
In industrial sectors such as manufacturing and infrastructure maintenance, technical text—including maintenance work orders (MWOs) and human-robot interaction logs—represents a critical repository of operational knowledge. However, this data is notoriously difficult to process due to domain-specific jargon, non-standard abbreviations (e.g., "DSHT Cons Thkner rplace bed press"), and inconsistent grammar. While modern techniques like weak supervision and crowdsourcing can rapidly generate labels for these large datasets, the resulting annotations are often riddled with errors. Traditional manual validation methods, such as scrolling through spreadsheets, are unscalable and inefficient. LabelVizier addresses this gap by providing a tool that allows domain experts to validate and relabel large-scale datasets efficiently.
The LabelVizier Workflow
LabelVizier operates through a three-phase, human-in-the-loop workflow designed to bridge the gap between automated labeling and expert domain knowledge.
- Error Profiling: The system begins by training a lightweight "surrogate" machine learning model (typically using TF-IDF and LinearSVC) to mimic existing, potentially flawed labels. Rather than aiming for perfect accuracy, this model serves to highlight areas of high uncertainty, providing "clues" through prediction probabilities that point toward likely labeling errors.
- Annotation Validation: This phase utilizes visual analytics to expose three primary error types: duplicate labels, wrong labels, and missing labels. By visualizing the surrogate model's behavior alongside the raw text, experts can identify systematic flaws in the initial labeling process.
- Annotation Relabeling: Once errors are identified, the system enables multi-scale relabeling. Users can apply fixes at the corpus level (global changes), sub-group level (clusters of similar records), or individual record level.
Interactive Visual Analytics
The system is implemented as an interactive notebook featuring several coordinated views to facilitate deep data exploration.
- Sunburst & Chord Diagrams: These views illustrate label hierarchies and co-occurrence patterns. Thick "chords" between labels often reveal semantic duplicates (e.g., "too hot" vs. "room too hot"), which are common when multiple annotators or weak supervision rules are used.
- Record Projection (t-SNE): This view clusters similar text records based on their semantic content. Users can color-code these clusters by "confidence score" to find incorrect labels or by "information density" to identify missing labels.
- Explainable AI (LIME): To build trust and provide context, the system uses LIME to explain the surrogate model's predictions. This helps experts determine if the model is correctly identifying domain concepts or mistakenly fixating on noise, such as HTML metadata.
Evaluation and Key Insights
LabelVizier was evaluated using two distinct datasets: an industrial HVAC maintenance dataset (21,000+ records) and a Natural Language Understanding (NLU) dataset (25,000+ records). Expert reviews from four professionals—including engineers and social scientists—highlighted several key findings:
- Efficiency and Speed: Experts found the tool significantly faster and more intuitive than traditional spreadsheet-based methods. The ability to relabel entire sub-groups using a lasso tool was particularly praised.
- The Power of Surrogate Models: The study demonstrated that a simple, lightweight model is sufficient to expose errors in complex datasets. It acts as a filter, allowing experts to focus their attention on the most suspicious data points.
- Expert Divergence: While ML-savvy users appreciated the complexity of t-SNE projections, domain-only experts preferred more guided, simplified interfaces. This suggests a need for adaptive UI designs based on user technical background.
Conclusion
LabelVizier demonstrates that a human-in-the-loop approach, combining lightweight machine learning with interactive visualization, can significantly improve the quality of technical text annotations. Future work will focus on lowering the learning curve for complex projections and adding more advanced filtering capabilities, such as Boolean operations, to further streamline the relabeling process.
Summary updated