TL;DR
This paper introduces MaD GUI, an open-source Python package designed to streamline the annotation and analysis of time-series data. By focusing on adaptability, usability, and user experience, MaD GUI aims to empower both domain experts and developers, significantly reducing the time and effort required for these crucial tasks in machine learning workflows.
What Problem Does This Paper Address?
Developing machine learning algorithms, particularly for time-series data, often relies on manual annotation to establish ground truth. This process is time-consuming and requires specialized tools. While existing graphical user interfaces (GUIs) for time-series annotation exist, they often suffer from a lack of reusability, poor adaptability to different data formats or algorithms, and insufficient attention to user experience. This leads to duplicated efforts, integration challenges, and a barrier to adoption, especially for domain experts without programming backgrounds. The authors aim to bridge this gap by providing a flexible, user-friendly, and open-source solution.
Methods
The core of MaD GUI is its plugin architecture, which allows for easy extension and customization. Developers can create plugins for:
- Importers: To load data from various recording devices and formats.
- Labels: To define annotation categories.
- Algorithms: To apply analysis methods and generate automated annotations.
- Exporters: To save processed data and annotations.
This architecture promotes composability, enabling the combination of different plugins at runtime. The software structure follows a Model-View-Controller (MVC) pattern to separate data representation, plotting, and user interaction, enhancing maintainability and understandability. For code quality, the package adheres to PEP8 guidelines and utilizes tools like black for formatting and pylint for additional checks. Cognitive complexity was assessed as a measure of code understandability.
The paper also details a small-scale study comparing MaD GUI with PALMS (a state-of-the-art Python package) across two main areas:
- Adaptability: Assessed by developers tasked with loading new data formats and implementing algorithms.
- User Interface: Evaluated by clinicians (domain experts without programming knowledge) performing annotation and analysis tasks.
Datasets
The paper does not explicitly mention datasets used for training MaD GUI. However, for the evaluation of the package's adaptability, the authors used data recorded using wearable sensors, specifically referencing data published by Weiss et al. [58] available via PhysioNet [59]. This data was used to test the ability of MaD GUI to handle formats not natively supported by the package.
Key Insights and Ideas
- Plugin Architecture is Key: The modular plugin system is the central innovation, allowing MaD GUI to be highly adaptable to diverse data sources and analytical needs without requiring core code modifications.
- Bridging the Gap: MaD GUI is designed to be accessible to both developers (through its Pythonic interface and clear documentation) and domain experts (through its intuitive GUI), a crucial step for translating research into practice.
- Usability Matters: The study highlights that even for developers, ease of use, clear documentation, and low barriers to entry are critical for software adoption, not just for end-users.
- Open-Source for Collaboration: By being open-source, MaD GUI fosters community contribution, allowing for continuous improvement and adaptation to new domains and use cases.
Key Results
The study yielded significant findings:
- Time Savings: MaD GUI demonstrated up to a 75% time saving for developers compared to using PALMS for tasks like loading data and implementing algorithms.
- Code Complexity: MaD GUI exhibits lower cognitive complexity and a higher comment-to-line ratio than PALMS, suggesting it is easier to maintain and extend.
- User Preference: Both developers and clinicians preferred MaD GUI in subjective ratings for usability and user experience.
- Adaptability: Developers found MaD GUI significantly easier to adapt for new data formats and algorithm integration, with tasks completed much faster.
- User Interface: Clinicians also found MaD GUI easier to learn and use for annotation tasks, with lower task completion times and higher usability scores (SUS: 85.0 for MaD GUI vs. 62.5 for PALMS).
Limitations and Future Work
- Small Sample Size: The study, while designed to identify major issues, used a small sample size, which might not capture all potential shortcomings.
- Developer Bias: Participants in the adaptability study were mostly from the same lab as the authors, potentially introducing bias. Efforts were made to mitigate this with external participants.
- Documentation: While generally praised, some participants found MaD GUI's documentation confusing in parts. The authors suggest separating documentation for different user types (developers vs. UI users) and improving tooltip visibility.
- Synchronization Issues: At the time of writing, known issues exist with video and sensor data synchronization for certain video types, with a fix in development.
- Future Work: The authors suggest improving usability and user experience further, for example, by more prominently displaying shortcuts and improving tooltip visibility. They also highlight the need for community use to reveal further documentation or usability issues.
Summary updated