CB+NN Ensemble to Improve Tracking Accuracy in Air Surveillance

Anoop Karnik Dasika, Praveen ParuchuriAAAI 2022 · OpenReview

Abstract

Finding or tracking the location of an object accurately is a crucial problem in defense applications, robotics, and computer vision. Radars fall into the spectrum of high-end defense sensors upon which global security and surveillance depend. The Indian Air Force utilizes a Multi-Sensor Tracking (MST) system to detect flights across India, developed by Bharat Electronics Limited (BEL). This paper describes a Machine Learning approach built on top of this existing legacy system. Trained on 13 million anonymized real-world radar data points, our approach increases tracking accuracy by 5 percent, from 91% to 96%. The methodology involves ensembling Neural Network and CatBoost models using Logistic Regression to predict and correct "Splitting" and "Merging" errors. Erroneous data points are corrected using cosine similarity to predict their original global tracks. The system has been transitioned to BEL and successfully tested in a simulation environment.

Introduction

Air surveillance relies on Multi-Sensor Tracking (MST) systems to fuse data from multiple radars into a single "Air Situation Picture." While traditional statistical methods like Kalman filters and Chi-square testing are industry standards, they often struggle with asynchronicity and overlapping sensor regions. These technical limitations lead to two critical types of errors:

  1. Splitting Errors ($E_s$): A single aircraft is incorrectly identified as multiple targets, potentially leading operators to believe an enemy flight is present when it is not.
  2. Merging Errors ($E_m$): Multiple aircraft are incorrectly identified as a single target, which can hide potential threats.

This research provides a Machine Learning "layer" that sits on top of existing systems to identify and correct these errors in real-time without requiring a total infrastructure overhaul.

Methodology

The proposed solution is a three-step pipeline that combines high-performance classifiers with a geometric correction step.

1. Ensemble Classification

For both splitting and merging errors, the system utilizes a Neural Network (parametric) and a CatBoost model (non-parametric). Because these models have different underlying logic, their error patterns are distinct. A Logistic Regression model acts as an ensemble manager, weighting the probabilities from both models to make a final prediction on whether an error has occurred.

2. Feature Engineering

The models utilize 54 features, including latitude, longitude, altitude, speed, and direction. To provide spatial context, the authors included $k$-means clustering of coordinates (at 10, 100, and 1000 centers), helping the model understand regional patterns in radar performance.

3. Correction via Cosine Similarity

Once a data point is flagged as an error, the system must rectify it. It analyzes the previous 1,000 data points and uses cosine similarity to find the "nearest neighbor" in the track history. It then re-assigns the erroneous point to the correct original global track.

The proposed AI methodology for real-time error detection and correction.

Datasets

The study utilized a massive real-world dataset provided by BEL, consisting of 16 million raw data points (13 million after cleaning). This dataset covers 12 days of continuous, anonymized multi-sensor tracking data from across Indian airspace.

Results and Discussion

The ensemble approach significantly outperformed individual models and traditional statistical methods. The primary insight was that ensembling CatBoost with a Neural Network yielded superior results because the two models "fail" in different ways, allowing the meta-learner to optimize the final output.

  • Tracking Accuracy: Improved from a baseline of 91% to 96% overall, with specific merging error tracking accuracy reaching 98.8%.
  • Model Comparison: As shown in the performance table, the "Ensemble - CatBoost with Neural Networks" consistently provided the highest accuracy and the best "Error Identification Accuracy" (the ability to specifically catch $E_s$ and $E_m$ cases).
Performance comparison of various algorithms for error detection and tracking accuracy.

Conclusion

This approach demonstrates the efficacy of using ML as a corrective wrapper for legacy defense systems. By addressing specific failure modes like splitting and merging errors, the system provides a more reliable Air Situation Picture. Future work will focus on addressing model drift over time and exploring geographic zonal specialization to further reduce error rates.

Summary updated

Abstract

Finding or tracking the location of an object accurately is a crucial problem in defense applications, robotics and computer vision. Radars fall into the spectrum of high-end defense sensors or systems upon which the security and surveillance of the entire world depends. There has been a lot of focus on the topic of Multi Sensor Tracking in recent years, with radars as the sensors. The Indian Air Force uses a Multi Sensor Tracking (MST) system to detect flights pan India, developed and supported by BEL(Bharat Electronics Limited), a defense agency we are working with. In this paper, we describe our Machine Learning approach, which is built on top of the existing system, the Air force uses. For purposes of this work, we trained our models on about 13 million anonymized real Multi Sensor tracking data points provided by radars performing tracking activity across the Indian air space. The approach has shown an increase in the accuracy of tracking by 5 percent from 91 to 96. The model and the corresponding code were transitioned to BEL, which has been tested in their simulation environment with a plan to take forward for ground testing. Our approach comprises of 3 steps: (a) We train a Neural Network model and a CatBoost model and ensemble them using a Logistic Regression model to predict one type of error, namely Splitting error, which can help to improve the accuracy of tracking. (b) We again train a Neural Network model and a CatBoost model and ensemble them using a different Logistic Regression model to predict the second type of error, namely Merging error, which can further improve the accuracy of tracking. (c) We use cosine similarity to find the nearest neighbour and correct the data points, predicted to have Splitting/Merging errors, by predicting the original global track of these data points.
BibTeX
@inproceedings{dasika2022cb,
  author = {Anoop Karnik Dasika and Praveen Paruchuri},
  title = {{CB+NN Ensemble to Improve Tracking Accuracy in Air Surveillance}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=0UF6lW8eXl},
}

Loading related content...

Figure