SLATE: A Sequence Labeling Approach for Task Extraction from Free-form Inked Content

Apurva Gandhi, Ryan Serrao, Biyi Fang, Gilbert Antonius, Jenna Hong, Tra My Nguyen, Sheng Yi, Ehi Nosakhare, Irene Shaffer, Soundararajan SrinivasanEMNLP 2022 · DOI

TL;DR

This paper introduces SLATE, a novel sequence labeling approach for automatically extracting actionable tasks from free-form, digitally handwritten notes. By jointly performing sentence segmentation and task classification in a single, low-latency model, SLATE significantly outperforms traditional two-model approaches and handles the unique challenges of the "inking" domain.

What Problem Does This Paper Address?

The shift to remote and hybrid work has increased reliance on virtual whiteboards for brainstorming and collaboration. Extracting actionable tasks from these free-form, often messy, handwritten notes is crucial for efficient workflow management. However, existing methods for task extraction are primarily designed for structured, typed text and struggle with the inherent challenges of inked content: poor grammar, inconsistent punctuation, typos, and the need for accurate sentence segmentation before classification. Furthermore, a two-step process (segmentation then classification) introduces latency, hindering real-time applications.

Methods

The core of SLATE is a single-model, sequence labeling approach. Instead of a pipeline of separate sentence segmentation and task classification models, SLATE uses a unified model. The authors propose three sequence labeling schemes:

  • Sentence-BI: A standard scheme for sentence segmentation, labeling tokens as "Beginning" or "Inside" of a sentence. This is used for their baseline.
  • SLATE-BIO: Designed for task extraction, labeling tokens as "Beginning," "Inside," or "Outside" of a task sentence.
  • SLATE-NTI: A novel multi-task learning scheme that simultaneously performs sentence segmentation and task sentence classification. It labels tokens as "Non-Task sentence beginning," "Task sentence beginning," or "Inside of a sentence." This is their flagship approach.

A key innovation is the incorporation of document layout metadata (line breaks and bullet points) as features. This helps the model overcome the lack of traditional linguistic cues (punctuation, capitalization) in inked content. They fine-tune a pre-trained RoBERTa encoder for this task.

Datasets

The authors introduce a novel dataset for task extraction from ink documents. This dataset consists of 200 vendor-created ink documents, featuring diverse content types (to-do lists, recipes, notes) and writing styles from over 50 donors. The raw ink data was processed through handwriting recognition and layout analysis, then manually annotated for sentence boundaries and task/non-task classification.

Key Insights and Ideas

  • Unified Model for Efficiency: Combining sentence segmentation and task classification into a single sequence labeling model drastically reduces latency compared to a two-model pipeline.
  • Layout Matters for Inked Text: Document layout features (like line breaks and bullet points) are surprisingly effective at compensating for the noisy and unstructured nature of handwritten notes, significantly improving segmentation performance.
  • Multi-task Learning for Better Segmentation: The SLATE-NTI scheme, which explicitly trains the model to segment both task and non-task sentences, leads to better overall segmentation quality than schemes focused solely on task sentences.
  • Context is Crucial for Task Identification: SLATE's ability to process entire text blocks at once (unlike the baseline's sentence-by-sentence approach) allows it to better leverage contextual information, leading to improved task classification, especially for tasks identified only due to context.

Key Results

SLATE-NTI with document metadata achieved a task F1 score of 84.4% and a boundary similarity score (B) of 88.4%. This represents a significant improvement over their baseline two-model approach, which achieved an F1 score of 82.3% and a B score of 85.3%. Crucially, SLATE achieved three times lower latency than the baseline. The paper also highlights that SLATE approaches significantly outperform the baseline in identifying tasks that are only discernible due to context.

Limitations and Future Work

The authors acknowledge that their dataset, while novel, is relatively small (around 200 documents). They suggest that with more data, language modeling could further enhance domain adaptation. Additionally, their current work focuses on recognized text from inked content, not the raw ink strokes themselves; handwriting recognition and layout analysis are treated as pre-processing steps. Future work could explore integrating these components more deeply or adapting the methodology to other free-form content types.

Summary updated

BibTeX
@inproceedings{gandhi2022slate,
  author = {Apurva Gandhi and Ryan Serrao and Biyi Fang and Gilbert Antonius and Jenna Hong and Tra My Nguyen and Sheng Yi and Ehi Nosakhare and Irene Shaffer and Soundararajan Srinivasan},
  title = {{SLATE: A Sequence Labeling Approach for Task Extraction from Free-form Inked Content}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-industry.21},
}
Figure