TL;DR
This paper details how the Swedish defense applied data mining and machine learning to analyze a massive influx of submarine intelligence reports, aiming to identify patterns and predict foreign submarine activity. By developing specialized database tools and applying techniques like evidential reasoning and genetic algorithms, they sought to make sense of ambiguous and often unreliable data, ultimately improving national security.
What Problem Does This Paper Address?
In the 1980s and early 1990s, Sweden faced a significant increase in suspected foreign submarine incursions into its territorial waters. This led to an overwhelming volume of intelligence reports, many of which were ambiguous, from civilian observers, and of varying credibility. The challenge was to systematically analyze this vast, complex, and often uncertain data to distinguish genuine threats from noise, identify patterns of activity, and ultimately predict future incursions. Traditional intelligence analysis methods struggled with the sheer scale and inherent ambiguity of the information.
Methods
The authors employed a multi-pronged approach:
- Specialized Database Technology (Cantor): Recognizing the limitations of standard relational databases, they utilized a system called Cantor, designed for efficient management, analysis, and visualization of large, spatio-temporal point observations. This allowed for more flexible data modeling and querying.
- Evidential Analysis (Dempster-Shafer Theory): To handle the ambiguity of reports that could refer to multiple submarines, they applied Dempster-Shafer theory. This framework uses the concept of "conflict" between reports to probabilistically assign them to different submarine "subsets," enabling separate analysis of each potential threat.
- Genetic Algorithms for Pattern Recognition: To make tactical predictions, they developed a machine-learning system using genetic algorithms. This system learns patterns from sequences of intelligence reports and uses these learned categories to predict future developments based on current scenarios.
- Statistical Analysis and Visualization: Various statistical tests were performed to determine if observations were random in time and space. Techniques like mapping observation density and creating spatio-temporal visualizations were used to highlight "hot spots" and potential patterns.
Datasets
The primary dataset consisted of approximately 6000 intelligence reports collected by Swedish defense authorities between 1981 and 1994. These reports were classified into four quality categories, along with "No submarine activity" and "Not decidable." The analysis focused on human observations, with a significant portion coming from civilian sightings.
Key Insights and Ideas
- The Power of Specialized Tools: Standard database systems were insufficient for the complex, spatio-temporal nature of submarine intelligence. A custom-built system like Cantor was crucial for effective analysis.
- Embracing Uncertainty: The data was inherently uncertain and often contradictory. Techniques like Dempster-Shafer theory were vital for managing this uncertainty and deriving meaningful insights rather than being paralyzed by it.
- Beyond Randomness: The analysis revealed that submarine activity was not random, suggesting deliberate patterns and areas of interest, even if the exact nature of the activity remained elusive.
- The Analyst's Role is Paramount: The paper emphasizes that mastering these advanced techniques is essential for the analyst group itself, not just consultants, to fully exploit the potential of data mining for top-secret information.
Key Results
- Statistical analysis showed that the observed submarine incursions were not random in time and space, rejecting the hypothesis of randomness.
- Cluster analysis, while requiring large datasets for definitive conclusions, provided valuable indications of spatio-temporal patterns.
- The evidential analysis approach allowed for the separation of reports into subsets, enabling more focused analysis of individual submarine activities.
- The genetic algorithm-based prediction system achieved a best-case accuracy of 54% for correct predictions, with a positional accuracy of 5 kilometers and a temporal accuracy of 48 minutes. While not perfect, this level of accuracy was deemed potentially very useful.
Limitations and Future Work
- Data Quality and Completeness: The paper acknowledges that the intelligence database contained information of varying credibility, and even with advanced analysis, many key questions remained unanswered. The commission also rejected the military's own classification scheme as insufficient.
- Definitive Conclusions: The authors note that while patterns were identified, drawing definitive conclusions about the number of credible observations and distinguishing them from other events was challenging.
- Prediction Accuracy: The achieved prediction accuracy, while useful, still leaves room for improvement. Future work could focus on refining these prediction models.
- Analyst Empowerment: The paper strongly advocates for analysts to be fully in charge of their tools and work, suggesting that the limited access and dedicated time available for the analysis only scratched the surface of what could be achieved.
Summary updated