ISOBlue HD: An Open-Source Platform for Collecting Context-Rich Agricultural Machinery Datasets

Yang Wang, He Liu, James V. Krogmeier, Amy R. Reibman, Dennis R. BuckmasterSensors 2020 · DOI

TL;DR

This paper introduces ISOBlue HD, an open-source platform for collecting rich, contextual data from agricultural machinery. By integrating CAN bus, GPS, and video streams, ISOBlue HD enables a deeper understanding of machine operations and farming logistics, crucial for advancing precision agriculture and automation.

What Problem Does This Paper Address?

Modern agriculture is increasingly reliant on complex machinery and data-driven decision-making. However, extracting meaningful insights from agricultural machinery data is challenging due to its proprietary nature and the difficulty in correlating raw sensor readings with actual operational context. Existing solutions often focus on single data streams or lack the ability to capture the nuanced "context" of machine operation (e.g., what the operator is doing, the machine's status, and its environment). This gap hinders the development of advanced applications like autonomous systems, improved decision support for operators, and more robust machine diagnostics. The authors aim to bridge this gap by providing a platform that collects and integrates diverse data streams with contextual information.

Methods

The core of the ISOBlue HD platform is a single-board computer (SBC) running a custom Linux-based Board Support Package (BSP). This SBC is powered directly from the machine's ISOBUS diagnostic port, which also provides access to the Controller Area Network (CAN) bus.

Key methodological components include:

  • Data Acquisition: The system simultaneously collects:
  • CAN Bus Data: Using two CAN transceivers at 250 kbps, capturing machine status and operational data.
  • GNSS Data: Via a USB GPS module providing positioning and timing information.
  • Video Data: From three IP cameras mounted inside the combine cab, capturing header status and operator actions.
  • Contextual Labeling: Video data is manually labeled using an open-source tool (MuViLab) to identify specific operator actions (e.g., joystick movements) and header states (e.g., up, down, transitioning). This labeling process is crucial for creating "context-rich" datasets.
  • Data Management: Apache Kafka is employed for robust data exchange and logging of CAN, GNSS, and diagnostic data. Video data is stored separately on a USB hard drive.
  • Power Management: A custom power management system, including a "can-watchdog" application and a relay PCB, allows the system to automatically suspend and wake up based on machine activity (detected via CAN bus signals), conserving power.
  • Remote Access: SSH port forwarding and a cellular modem enable remote diagnostics and data streaming.

Datasets

The paper introduces a newly collected "context-rich" dataset from a combine harvester during a 2019 wheat harvest. This dataset comprises:

  • 69 million CAN frames
  • 230,000 GPS points
  • 437 GB of video data

This dataset was collected over 84 hours of harvest time and is unique in its integration of machine telemetry (CAN), precise location (GPS), and visual context (video) with associated contextual labels. The authors do not explicitly mention using existing datasets for training or evaluation, focusing instead on the creation and initial analysis of their novel dataset.

Key Insights and Ideas

  • Context is King: The paper strongly emphasizes that raw sensor data (CAN, GPS) becomes significantly more valuable when fused with contextual information derived from video and operator actions. This contextualization allows for a much deeper understanding of machine behavior and operational logistics.
  • Open-Source for Agriculture: The development of an open-source platform like ISOBlue HD is crucial for democratizing access to advanced agricultural data collection and analysis tools, fostering wider research and development in precision agriculture.
  • CAN Data is Rich, but Opaque: While CAN bus data contains a wealth of information, its proprietary nature makes direct interpretation difficult. The paper demonstrates how contextual labels can be used to infer the meaning of unknown CAN IDs, potentially enabling reverse-engineering and non-proprietary analysis.
  • Operator Actions Matter: The video data, specifically capturing operator inputs, reveals that human intervention is still a significant factor in modern farming. Understanding these actions in conjunction with machine data can highlight areas for automation or operator assistance to reduce fatigue and improve safety.
  • Data Fusion for Deeper Insights: By combining GPS tracks with contextual labels, the authors show how to differentiate between harvest areas, transitions, and headland maneuvers, providing a more nuanced view of field operations than GPS data alone.

Key Results

  • The ISOBlue HD platform was successfully deployed on a combine harvester, collecting a substantial dataset (437 GB) over 84 hours of operation.
  • Analysis of the collected data demonstrated that contextual knowledge could be inferred regarding harvest logistics (paths, speeds, header status, material transfer) and sensor data semantics.
  • Preliminary analysis of contextual labels revealed that "header down" is the most frequent header position, and "joystick 1" (header height/tilt) and "joystick 2" (reel height) are the most frequently used operator controls, indicating consistent manual adjustments during harvest.
  • The paper successfully demonstrated how contextual labels could be used to interpret otherwise opaque CAN signals, suggesting potential flow or yield sensor readings based on correlations with operator actions and header status.

Limitations and Future Work

Limitations:

  • Enclosure Bulkiness: The current enclosure for the hardware is described as bulky and inconvenient for in-cab deployment.
  • Inconsistent Internet Access: Sporadic network connectivity in rural areas led to delayed data streaming and failed remote login attempts, highlighting the need for more robust data streaming policies.
  • Kafka as a Database: While Kafka is effective for data exchange, fetching data for post-processing was cumbersome, suggesting that a more traditional database might be better suited for indexed data logging and querying.

Future Work:

  • Enclosure Optimization: Developing smaller, more integrated, and potentially cabless enclosures.
  • Intelligent Data Streaming: Implementing policies to batch data during poor network conditions and send it opportunistically.
  • Database Replacement: Exploring more compact databases that offer both indexed logging and querying capabilities.
  • Scalability and Networked Sensing: Deploying multiple ISOBlue HD units on different machines to create a distributed sensing network for real-time context inference and sharing.
  • Data Compression and Efficiency: Developing efficient data transmission schemes, including proper compression and error control, to reduce bandwidth usage, especially for CAN data.
  • Automation and AI: Leveraging computer vision and AI to automatically label video data and automate tasks like header height adjustment, improving safety and operational effectiveness.
  • Comprehensive Data Pipeline: Building a complete data pipeline that integrates GNSS track clustering, CAN signal interpretation, and sensor signal classification.

Summary updated

Abstract

This paper introduces an open-source platform called <i>ISOBlue HD</i> for acquisition of context-rich data from agricultural machinery. We call these datasets context-rich, because they enable the identification of machine status and farming logistics by properly labeling, fusing, and processing the data. The system includes a single board computer, a cellular modem, local storage, and power-over-ethernet switch to sensors. The system allows remote diagnostics and access, automatic startup/shut down with vehicle operations, and uses Apache Kafka to enable robust data exchange. <i>ISOBlue HD</i> was deployed in a combine harvester during a 2019 wheat harvest for simultaneously capturing 69 million CAN frames, 230,000 GPS points, and 437 GB of video data, focusing on header status and operator actions over 84 h of harvest time. Analyses of the collected data demonstrate that contextual knowledge can be inferred on harvest logistics (paths, speeds, header status, material transfer) and sensor data semantics.
BibTeX
@inproceedings{wang2020isoblue,
  author = {Yang Wang and He Liu and James V. Krogmeier and Amy R. Reibman and Dennis R. Buckmaster},
  title = {{ISOBlue HD: An Open-Source Platform for Collecting Context-Rich Agricultural Machinery Datasets}},
  booktitle = {Sensors},
  year = {2020},
  doi = {10.3390/s20205768},
}
Figure