DNA: Denoised Neighborhood Aggregation for Fine-grained Category Discovery

Wenbin An, Feng Tian, Wenkai Shi, Yan Chen, Qinghua Zheng, Qianying Wang, Ping Chen

Introduction

The challenge of Fine-grained Category Discovery under Coarse-grained supervision (FCDC) addresses a critical bottleneck in machine learning: the high cost of expert labeling. While it is relatively simple to categorize data into broad domains (e.g., "Birds"), identifying specific sub-categories (e.g., "Swinhoe's Pheasant") requires significant domain expertise. Existing self-supervised methods often rely on instance-level contrastive learning, which treats every data point as a unique class. While effective for general representation learning, this approach often fails to capture the semantic nuances required to distinguish between highly similar fine-grained categories, as it lacks a mechanism to encourage cluster compactness.

To address these limitations, we present DNA (Denoised Neighborhood Aggregation), a framework that leverages coarse labels to discover fine-grained structures by aggregating "clean" local neighborhoods.

Methodology: Denoised Neighborhood Aggregation

The fundamental premise of DNA is that a query’s $k$-nearest neighbors ($k$-NN) in the embedding space should ideally belong to the same fine-grained category. However, in practice, $k$-NN retrieval is noisy; many neighbors may share a coarse label but belong to different sub-categories. DNA introduces three specific "denoising" constraints to refine these neighborhoods before they are used for model optimization.

Neighborhood Refining Principles

  1. Label Constraint: This principle utilizes the available coarse-grained supervision. Any neighbor that does not share the same coarse label as the query is immediately discarded, ensuring that the search space is restricted to semantically relevant domains.
  2. Reciprocal Constraint: To filter out outliers and handle long-tail distributions, DNA enforces a bidirectional check. A neighbor is only considered "reliable" if the query also appears in that neighbor’s own $k$-nearest neighbor list.
  3. Rank Statistic Constraint: Recognizing that raw cosine similarity can be misleading in high-dimensional spaces, DNA employs rank statistics. By comparing the indices of the top dimensions in feature vectors, the model identifies samples that share the same underlying feature importance, which is a more robust indicator of fine-grained similarity.

The Generalized EM Framework

The DNA framework is structured as a Generalized Expectation-Maximization (EM) algorithm, facilitating an iterative improvement of both the representations and the discovered clusters:

  • E-step (Neighborhood Retrieval): The model retrieves and refines reliable neighbors from a dynamic queue using the three constraints mentioned above.
  • M-step (Neighborhood Aggregation): The model parameters are optimized to pull these refined neighbors closer together in the embedding space, thereby increasing cluster density and separation.

Experimental Evaluation

The DNA framework was evaluated on three benchmark datasets for intent detection and text classification: CLINC (150 fine-grained intents), WOS (33 categories), and HWU64 (64 intents).

Performance Gains

DNA established a new state-of-the-art (SOTA) performance across all benchmarks. On average, the model improved Accuracy (ACC) by 9.96% over the previous leading method, WSCL. The most significant gain was observed on the CLINC dataset, where accuracy increased by 13.64%.

Retrieval and Visualization

The effectiveness of the denoising principles is evidenced by a 21.31% improvement in the accuracy of retrieved neighbors compared to standard $k$-NN. Furthermore, t-SNE visualizations indicate that DNA produces significantly more compact and well-separated clusters than previous instance-level methods, which often leave fine-grained categories entangled.

Discussion and Insights

The success of DNA highlights the importance of the Compactness Hypothesis in fine-grained discovery. By prioritizing the formation of tight clusters and moving away from the "push-everything-away" nature of instance-level contrastive learning, the model creates clearer decision boundaries.

A key technical insight is the robustness of Rank Statistics. In high-dimensional data, standard similarity metrics often struggle with noise. By focusing on the ranking of feature dimensions, DNA effectively filters out irrelevant feature components that might otherwise lead to false-positive neighbor assignments.

Conclusion

DNA demonstrates that denoising local neighborhoods is a powerful strategy for discovering fine-grained categories under coarse supervision. While the approach introduces some memory and computational overhead due to the dynamic queue and sorting operations, the substantial gains in accuracy and cluster quality represent a significant step forward for self-supervised discovery. Future research may focus on scaling these denoising principles to larger datasets and exploring their applicability in computer vision tasks.

Summary updated

BibTeX
@inproceedings{an2023dna,
  author = {Wenbin An and Feng Tian and Wenkai Shi and Yan Chen and Qinghua Zheng and Qianying Wang and Ping Chen},
  title = {{DNA: Denoised Neighborhood Aggregation for Fine-grained Category Discovery}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.756},
}
Figure