Learning Uniform Semantic Features for Natural Language and Programming Language Globally, Locally and Sequentially

Yudong Zhang, Wenhao Zheng, Ming Li

Introduction

In software engineering, practitioners frequently navigate between natural language (NL)—found in documentation, bug reports, and forum discussions—and programming language (PL), the executable source code. Bridging the semantic gap between these two modalities is critical for tasks such as code retrieval, duplicate detection, and automated documentation. Traditionally, models have treated NL and PL as distinct entities, often failing to capture the shared underlying logic. UniEmbed, a framework presented at AAAI-19, addresses this by projecting both modalities into a single, uniform semantic space.

The Challenge of the Semantic Gap

Existing software mining methods often struggle with two primary issues. First, they frequently rely on incomplete information, focusing solely on lexical tokens or rigid syntax while ignoring broader sequential logic or localized conceptual units (e.g., specific algorithmic patterns). Second, the "semantic gap" prevents models from recognizing that a natural language phrase like "sort an array" and a Java implementation of QuickSort represent the same concept.

The UniEmbed Framework

UniEmbed utilizes two parallel neural pipelines—Textnet for natural language and Codenet for programming language—to extract features across multiple dimensions.

The UniEmbed architecture featuring parallel Textnet and Codenet pipelines with multi-level feature extraction.

The framework operates through four distinct layers:

  1. Encoding Layer: Tokens are converted into 300-dimensional vector representations using pre-trained GloVe embeddings.
  2. Multi-info Layer: This core component extracts three levels of semantic information:
  3. Global Information: Captures the overall intent by averaging term meanings across the entire snippet.
  4. Local Information: Uses Convolutional Neural Networks (CNNs) to identify patterns within small regions. For code, this is hierarchical, capturing both individual statements and inter-statement interactions.
  5. Sequential Information: Employs Gated Recurrent Units (GRUs) to model logical flow. In the Codenet pipeline, this is split into term-level and statement-level processing to respect the structure of code.
Examples of global, sequential, and local information extracted from code snippets to illustrate the multi-info layer's focus.
  1. Fusion & Attention Layer: Rather than simple concatenation, an attention mechanism dynamically weights the importance of global, local, and sequential features based on the specific task.
  2. Uniform Space Projection: Using a Triplet Loss function, the model ensures that semantically similar pairs (e.g., a code snippet and its corresponding comment) are positioned closely in the vector space, while dissimilar pairs are pushed apart.

Hierarchical Code Processing

A key insight of UniEmbed is that code is inherently hierarchical. Unlike natural language sentences, code possesses "atomicity," where a single statement serves as a fundamental unit of meaning. By processing code at both the term and statement levels, UniEmbed captures structural nuances that flatter models miss. For instance, the sequential order of a modification followed by a sort operation (Sequential Information) defines the final state of an array, a distinction the model is designed to recognize.

Experimental Results

The researchers evaluated UniEmbed on three large-scale datasets from StackOverflow: dup-question (duplicate detection), code-anno (code annotation), and code-clone (functional similarity).

  • Duplicate Question Retrieval: UniEmbed achieved an AUC of 95.44%, significantly outperforming traditional attentive pooling models (72.23%).
  • Code Clone Detection: The model reached an 87.69% AUC on Type-4 clones—snippets that are functionally identical but syntactically diverse—demonstrating its ability to grasp deep semantics.
  • Code Annotation: UniEmbed outperformed baselines by over 13% in AUC, proving that the model successfully bridged the "semantic gap" between text and code.

Conclusion and Future Directions

UniEmbed demonstrates that joint training on NL and PL tasks allows for a more robust understanding of software semantics. While currently optimized for Java and reliant on GloVe embeddings, the framework provides a general-purpose foundation for cross-modal software mining. Future work may involve integrating code-specific embeddings like Code2Vec or adapting the architecture for more diverse programming paradigms.

Summary updated

Abstract

Semantic feature learning for natural language and programming language is a preliminary step in addressing many software mining tasks. Many existing methods leverage information in lexicon and syntax to learn features for textual data. However, such information is inadequate to represent the entire semantics in either text sentence or code snippet. This motivates us to propose a new approach to learn semantic features for both languages, through extracting three levels of information, namely global, local and sequential information, from textual data. For tasks involving both modalities, we project the data of both types into a uniform feature space so that the complementary knowledge in between can be utilized in their representation. In this paper, we build a novel and general-purpose feature learning framework called UniEmbed, to uniformly learn comprehensive semantic representation for both natural language and programming language. Experimental results on three real-world software mining tasks show that UniEmbed outperforms state-of-the-art models in feature learning and prove the capacity and effectiveness of our model.
BibTeX
@inproceedings{zhang2019learning,
  author = {Yudong Zhang and Wenhao Zheng and Ming Li},
  title = {{Learning Uniform Semantic Features for Natural Language and Programming Language Globally, Locally and Sequentially}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33015845},
}
Figure