FAST: Fast Annotation tool for SmarT devices

Shunyo Kawamoto, Yu Sawai, Kohei Wakimoto, Peinan ZhangEMNLP 2021

TL;DR

Annotating data for machine learning tasks is crucial but often cumbersome, especially on mobile devices. This paper introduces FAST, a novel web-based annotation tool designed with a mobile-first user experience in mind. FAST significantly speeds up annotation tasks and improves usability compared to existing tools, making crowd-sourced annotation more efficient and accessible.

What Problem Does This Paper Address?

In real-world applications, particularly in areas like sentiment analysis or evaluating natural language generation (NLG) outputs, it's essential to have a diverse group of annotators. However, a significant portion of users interact with applications via mobile devices. Existing annotation tools, while functional, often fail to optimize the user experience (UX) for these mobile users. They might simply adapt desktop interfaces to mobile screens, leading to inefficiencies and frustration. This paper addresses the gap of specialized, mobile-centric annotation tools that prioritize ease of use and speed for a broad range of annotators, not just domain experts.

Methods

The core of FAST is its focus on a flexible, customizable web application designed for mobile-first UX. Key methodological aspects include:

  • Flexible Interface: FAST is built as a web application, allowing it to run on any device with a browser. Its interface is highly customizable, enabling issuers to design views optimized for specific tasks.
  • Card UI and Multi-label UI: FAST offers two primary UI types:
  • Card UI: Inspired by popular mobile apps like Tinder, this UI presents information on a "card" and allows users to respond with a single swipe or tap. This single-action approach is designed for speed and intuitiveness.
  • Multi-label UI: This UI presents multiple buttons for annotators to select from, suitable for tasks with more complex choice sets.
  • Serverless Architecture: FAST utilizes Google Cloud Platform and Firebase for its backend. This serverless approach reduces management costs and allows for convenient operation without the need for constant updates or server maintenance.
  • Detailed Logging: The system collects comprehensive logs, including user actions, timestamps, device information, and screen size. This data is crucial for analyzing annotation difficulty, user behavior, and overall efficiency.
  • API Integration: FAST supports data communication via APIs, allowing developers to integrate annotation workflows seamlessly into their existing pipelines for issuing and analyzing annotations.

Datasets

The paper does not introduce new datasets for training. Instead, it uses product review data from an e-commerce site for its evaluation experiments. This data consisted of text with 50 or fewer Japanese characters and a five-point rating score. For the specific experiment, the data was simplified to a binary classification task (satisfactory/unsatisfactory) by filtering out intermediate ratings.

Key Insights and Ideas

  • Mobile-First UX is Key: The study strongly suggests that designing annotation tools with a mobile-first approach significantly enhances user experience and efficiency, even for tasks that could be performed on a desktop.
  • Single-Action UI for Speed: The Card UI's single-action design (swiping or tapping) dramatically reduces the number of steps required per annotation, leading to substantial time savings compared to multi-step interfaces.
  • Serverless is Cost-Effective: The serverless architecture proves to be a cost-efficient solution for annotation tools, especially for applications with fluctuating or low-frequency access.
  • Usability Trumps Raw Power (Sometimes): While multi-label UIs offer more choice, the Card UI's superior usability and speed on mobile devices make it a more effective choice for many common annotation tasks.
  • Mobile Annotators Work "On the Go": The experiment confirmed that mobile annotators often utilize spare time (e.g., during commutes) for annotation, highlighting the flexibility and accessibility FAST provides.

Key Results

  • Annotation Efficiency: FAST, particularly with its Card UI on mobile devices, reduced annotation time by approximately 43% compared to existing tools. The average annotation time for FAST's Card UI on mobile was 4.7 seconds, compared to 8.3 seconds for existing tools.
  • Annotation Quality: The study found that FAST maintained high annotation quality, with accuracy rates consistently around 0.97 and inter-annotator agreement (Krippendorff's α) around 0.95-0.96 for the Card UI.
  • Qualitative Usability: The Card UI on mobile devices received the highest rankings for usability, with an average ranking of 2.03, significantly better than other configurations. Annotators reported that mobile devices allowed them to work in any location, confirming the tool's flexibility.
  • PC vs. Mobile Performance: The performance (speed and quality) of PC and mobile devices was found to be comparable, suggesting that mobile devices are a viable and effective platform for annotation.

Limitations and Future Work

  • Limited Experiment Duration: The study acknowledges that the five-day experiment duration might not be sufficient to fully confirm the long-term efficiency benefits of mobile devices, especially concerning annotator fatigue.
  • Simplified Task: The evaluation focused on a binary classification task. While the Card UI can be adapted for more complex labels, further research is needed to explore its performance in those scenarios.
  • Future Work: The authors plan to:
  • Conduct more detailed quantitative evaluations using metrics like user operation trajectories and NASA TLX for task load.
  • Expand the range of supported task types.
  • Enhance management functions.
  • Develop support for on-premise environments to broaden its applicability as a general-purpose annotation tool.

Summary updated

BibTeX
@inproceedings{kawamoto2021fast,
  author = {Shunyo Kawamoto and Yu Sawai and Kohei Wakimoto and Peinan Zhang},
  title = {{FAST: Fast Annotation tool for SmarT devices}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
}
Figure