Abstract
Traditional "Data-to-Decision" (D2D) pipelines in high-stakes environments like disaster relief and security often suffer from a rigid, one-way flow of information. This paper proposes a transition toward "Conversational Sensing," where humans and machine agents interact through a dynamic, peer-to-peer network. By leveraging Controlled Natural Language (CNL), specifically ITA Controlled English (CE), the authors demonstrate a framework where untrained human observers can provide structured sensing data that machines can immediately process, fuse, and explain.
The Problem: Rigid Data Pipelines
In fields such as emergency response and policing, the standard D2D process is typically a linear progression: sensors collect raw data, analytic services process it, and a human decision-maker eventually acts upon the results.
This model faces two primary hurdles. First is the Semantic Gap: human eyewitnesses naturally communicate in rich, nuanced natural language (NL), but automated reasoning and information fusion systems require formal, structured data to function. Second is the issue of Rigid Workflows: traditional pipelines are primarily "data-driven" (forward-chaining). They lack the flexibility for "backward-chaining"—where a commander's specific information need can dynamically task sensors or prompt humans for clarification to fill a specific intelligence gap.
Proposed Solution: Conversational Sensing
The authors propose transforming this pipeline into a conversational network. In this model, humans and machines query each other, clarify reports, and explain reasoning in real-time. This creates a collaborative environment rather than a series of isolated silos.
ITA Controlled English (CE)
The linchpin of this approach is ITA Controlled English (CE). CE is a Controlled Natural Language—a subset of English with restricted syntax. It is designed to be perfectly readable by humans while simultaneously serving as a formal logic representation for machine processing. By using CE as the "common ground," the system eliminates the need for complex translation layers, making the entire process transparent and auditable for human supervisors.
Interaction Protocol
The framework defines four core interaction types to facilitate this dialogue:
- Confirm: This process takes a messy natural language report (e.g., "Black car heading south") and refines it into a precise, machine-readable CE statement.
- Ask/Tell: These are standard query-and-response exchanges between agents, allowing for information retrieval and sharing.
- Gist/Expand: To avoid overwhelming users with verbose formal logic on small screens, the system provides a "gist" (summary) while allowing users to "expand" and view the underlying logic if needed.
- Why: This crucial feature enables humans to request the rationale behind a machine's inference, such as why a specific vehicle was flagged as suspicious, fostering trust in automated systems.
Experimental Evaluation
The authors implemented two prototype agents: Moira, a mobile application for human reporting, and Sam, an agent that matches mission requirements to available sensors like drones.
Pilot Study
A pilot experiment was conducted with 20 UK undergraduate students acting as untrained observers. Participants were shown images of emergency service scenes—such as police on horseback, fire engines, and ambulances—and asked to describe them using the Moira app.
The system's performance was evaluated based on its ability to extract valid CE entities and relationships from these descriptions. On average, the agent successfully extracted 2.3 model elements (concepts, instances, or properties) per description, demonstrating that even simple algorithms can bridge the gap between natural language and formal logic.
Key Insights
- Gamification of Sensing: Interestingly, the authors found that providing users with a "score" based on the machine's understanding encouraged more detailed and elaborate descriptions. This suggests that feedback loops can significantly improve the quality of human-provided data by incentivizing clarity.
- The "Gist" Concept: Recognizing that raw code or verbose CNL is impractical for mobile screens, the authors utilized "gist" forms—simplified text or generated icons—to maintain a natural interaction flow while preserving formal logic underneath.
Results and Future Work
The prototype achieved a System Usability Scale (SUS) score of 64, a promising baseline for early-stage research. Despite using a relatively simple "bag of words" approach, the agent effectively gathered usable formal information from untrained subjects.
Future work aims to address current limitations, such as improving the handling of complex linguistic features like anaphora and integrating multimodal data (audio, video, GPS). By bridging the gap between human intuition and machine analytics, Conversational Sensing paves the way for more agile and transparent decision-making in critical scenarios.
Summary updated