Joint Optimization of Data Structure and Anomaly Detection

This direction is split into other directions: joint clustering and anomaly detection. Its last version stays readable and citable here.

Version 1 of 143 papers62 conceptsfrozen 2026-10-04

In traditional data science pipelines, clustering and anomaly detection are treated as sequential, independent steps. You either "clean" the data by removing outliers before finding patterns, or you find the patterns first and flag whatever doesn't fit.

However, as noted by Linsey Pang and Fabio Ramos in Integer Programming Relaxations for Integrated Clustering and Outlier Detection, this creates a circular dependency. Outliers are defined by their deviation from a "normal" structure, but that structure cannot be accurately defined if outliers are allowed to distort it. If you cluster first, the outliers pull the cluster centers toward themselves; if you detect outliers first, you might accidentally prune valid data points that simply belong to a yet-to-be-discovered small cluster.

The research direction emerging from these papers moves away from sequential pipelines toward joint optimization frameworks. By treating the discovery of structure and the identification of anomalies as a single, unified problem, these models create a mutually reinforcing feedback loop.

The Feedback Loop of Mutual Reinforcement

The core intuition behind this research is that "inlierness" and "structure" are two sides of the same coin. In a unified framework, the model's confidence in a cluster's shape helps it identify outliers, and the removal of those outliers immediately sharpens the cluster's definition.

Zeyu Fang and Ming Gu demonstrate this in Towards a Unified Framework of Clustering-based Anomaly Detection with UniCAD. They treat cluster centers as "gravitational masses." As the model trains, data points are pulled toward these centers. Points that resist this pull—those with a high "resultant force" in the opposite direction—are flagged as anomalies. Crucially, by identifying these anomalies during training, the model prevents them from "polluting" the gravitational pull of the normal clusters, leading to more stable and accurate cluster centers.

This same logic applies to supervised tasks. In Learning to classify with possible sensor failures, Alfred O. Hero and colleagues introduce GEM-MED, which integrates anomaly detection directly into the classification objective. Instead of letting corrupted samples (like those from a failing sensor) become influential "support vectors" that warp the decision boundary, the model identifies them as outliers and ignores them, resulting in a cleaner, more robust classifier.

Contextual Anomalies in Networks

The power of joint optimization is most visible in complex information networks. A "community outlier" isn't just a node that looks weird globally; it’s a node that behaves normally in general but appears anomalous when compared to its specific social or functional neighborhood.

In On community outliers and their efficient detection in information networks, Feng Liang and Jing Gao propose CODA. By simultaneously discovering communities and detecting outliers, CODA ensures that the "normal" baseline for any node is defined by its actual peers. Similarly, Rose Yu and Yan Liu developed GLAD (GLAD: group anomaly detection in social media analysis) to find coordinated "group anomalies." Because the model infers group membership and detects deviations concurrently, it can spot subtle malicious behaviors—like a botnet—that would be invisible if you only looked at individual nodes or pre-defined clusters.

Rectification Instead of Deletion

Recent work has pushed this idea even further: if we can identify an outlier and the structure it belongs to simultaneously, why not "fix" the outlier?

Jose Blanchet and Jiajin Li explore this in Automatic Outlier Rectification via Optimal Transport. Rather than simply discarding noisy data, their framework uses Optimal Transport to surgically relocate outliers back toward the main distribution. This "rectification" is informed by the downstream task, ensuring that the data cleaning process is directly optimized to improve the final model's performance.

Why It Works: Breaking the Circularity

The success of these unified approaches across domains—from time series forecasting (François-Xavier Aubet and Jan Gasthaus in Monte Carlo EM for Deep Time Series Anomaly Detection) to fair clustering (Hanyu Song and Hongfu Liu in Deep Clustering based Fair Outlier Detection)—stems from three core advantages:

  1. Stability: By filtering noise during the learning of the structure, the resulting "normal" model is not biased by the very anomalies it is trying to find.
  2. Precision: Anomalies are evaluated against the most relevant local context (the emerging cluster or community) rather than a global average.
  3. Efficiency: Joint optimization prevents the propagation of errors. In a two-stage pipeline, if the first stage (e.g., clustering) fails due to noise, the second stage (anomaly detection) is doomed. Unified models allow the two tasks to negotiate a solution that satisfies both structural consistency and outlier sparsity.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_joint_optimization_of_data_structure_and_2ef8c966,
  title = {Joint Optimization of Data Structure and Anomaly Detection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/joint-optimization-of-data-structure-and-anomaly-detection/rdr_2ef8c966908c4490ac896d84b683a254},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_joint_optimization_of_data_structure_and_2ef8c966_v1,
  title = {Joint Optimization of Data Structure and Anomaly Detection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/joint-optimization-of-data-structure-and-anomaly-detection/rdr_2ef8c966908c4490ac896d84b683a254/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}