SmartCS: Enabling the Creation of ML-Powered Computer Vision Mobile Apps for Citizen Science Applications without Coding

Fahim Hasan Khan, Akila de Silva, Gregory Dusek, James Davis, Alex PangarXiv 2024 · DOI · arXiv

Introduction

Citizen science represents a transformative approach to large-scale data collection, leveraging public participation to advance scientific research. However, the effectiveness of these initiatives is often hampered by two primary challenges: data quality and technical barriers. Non-expert participants frequently struggle to accurately identify complex subjects in the field, and the cost of developing custom mobile applications with integrated machine learning (ML) guidance is often prohibitive, requiring specialized programming expertise and significant financial investment.

SmartCS addresses these hurdles by providing an open-source, no-code platform that empowers researchers and students to build mobile citizen science apps with integrated, offline-capable ML. By focusing on client-side inference, SmartCS ensures that high-quality computer vision tools remain functional in remote areas where internet connectivity is unavailable.

Comparison of server-side vs. client-side ML architectures. SmartCS utilizes client-side integration to ensure functionality in remote areas without network connectivity.

The SmartCS Workflow

The platform streamlines the development process through a three-step automated workflow designed for users with no prior coding experience.

The complete SmartCS workflow, encompassing dataset creation, ML model training, and mobile app building.

1. Dataset Creation

The workflow begins with data acquisition and preparation. SmartCS provides integrated tools for extracting individual frames from video data and facilitates integration with open-source labeling software such as AnyLabeling or LabelImg. The platform automatically handles the formatting and splitting of data into training, testing, and evaluation sets, ensuring a robust foundation for model development.

2. ML Model Training

To optimize performance on mobile hardware, SmartCS utilizes transfer learning on lightweight, mobile-optimized models from the TensorFlow Lite library. The platform supports "one-stage" detectors like SSD and YOLO, which prioritize inference speed—a critical factor for real-time mobile applications. Users have the flexibility to train models locally or utilize cloud-based resources like Google Colab to access free GPU acceleration.

3. Mobile App Building

The final stage involves the assembly of the mobile application. SmartCS offers GUI-based templates for both iOS and Android, allowing users to customize visual elements such as colors and logos. The trained ML model is integrated into the app without any manual coding. Furthermore, the system incorporates Fastlane to automate the deployment process to digital distribution platforms.

Applications and Case Studies

The versatility of SmartCS has been demonstrated through several specialized applications:

  • RipSnap: Designed to identify rip currents using a dataset of 3,360 labeled images sourced from drone footage and beach webcams.
  • Recycle This: An app for identifying household recyclables, trained on 2,500 images from various public datasets.
  • TidalNow: Focused on marine biology, this app identifies five different tide pool species using a dataset of approximately 600 images per class.

Key Insights: Democratizing Development

A significant finding of this research is the platform's ability to democratize app development. In pilot studies, high school students with no prior programming experience were able to design and build functional, research-grade ML applications within just one to two weeks.

Beyond simple data collection, the platform utilizes ML as an educational tool. Real-time bounding boxes provide immediate feedback to users, helping them learn to recognize specific subjects (e.g., a particular shell or a rip current) independently over time. For projects starting without existing data, the "Expert Mode" allows for initial raw data collection by volunteers, which is then curated by experts to train subsequent iterations of the ML model.

Results and Performance

Benchmarking of various model architectures revealed that YOLOv8m provided the highest precision (50.2 mAP) with an inference speed of 32ms. However, EfficientDet D2 was noted for offering the most stable performance across a wide range of modern smartphone hardware.

User satisfaction surveys indicated high levels of approval. App creators (high school students) specifically praised the platform for being free and requiring no programming skills, both scoring approximately 4.5/5 on a Likert scale. Beta testers of the resulting apps also reported high levels of satisfaction with the ML capabilities and overall utility of the tools.

Limitations and Future Directions

While successful, the current iteration of SmartCS has areas for improvement. User feedback suggested that the UI templates are currently too simplistic; future updates will include more diverse and aesthetically refined designs. Additionally, the authors plan to implement a "human-in-the-loop" feedback system, allowing users to correct ML errors in real-time. These corrections will then be used to automatically retrain and improve the models, creating a seamless cycle of refinement. As mobile hardware continues to evolve, the platform will also incorporate more advanced ML architectures into its no-code pipeline.

Summary updated

BibTeX
@misc{khan2024smartcs,
  author = {Fahim Hasan Khan and Akila de Silva and Gregory Dusek and James Davis and Alex Pang},
  title = {{SmartCS: Enabling the Creation of ML-Powered Computer Vision Mobile Apps for Citizen Science Applications without Coding}},
  year = {2024},
  eprint = {2405.14323},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.14323},
}
Figure