CRUSTPairs: A Cross-Language Dataset of Paired C and Rust Solutions

Vojtech Hejlek, Leonardo Jesus Almeida, Alneu de Andrade LopesarXiv 2026 · arXiv

TL;DR

CRUSTPairs is the first dataset of aligned C and Rust solutions at the whole-program level, designed to benchmark cross-language retrieval and similarity. The authors demonstrate that while pretrained code models often rely on superficial variable name matching, finetuning on this dataset allows models to learn deep structural mappings that remain robust even when all identifiers are removed.

What Problem Does This Paper Address?

Memory safety vulnerabilities are a persistent plague in systems programming, with Microsoft and Android attributing roughly 70% of their CVEs to this class of defect. The industry is aggressively moving toward Rust to mitigate these risks, but migrating legacy C codebases is a monumental task.

While automated translation tools are emerging, evaluating them is tricky. Current benchmarks focus on whether the output compiles and passes tests. However, legacy code often lacks comprehensive test suites. This creates a need for a different kind of evaluation: semantic similarity. We need to be able to judge if a Rust program is a faithful rendering of a C source without relying solely on execution. Before CRUSTPairs, no dataset existed that paired these two languages at the level of complete, functional solutions.

Methods

The authors constructed CRUSTPairs by pulling accepted solutions for the same competitive programming problems from two major archives (CodeNet and xCodeEval). To ensure the models learn meaningful patterns rather than just "cheating" by looking at comments or specific function names, they applied a rigorous preprocessing pipeline:

  • Normalizing: Stripping comments, removing unused code, and applying standard formatters.
  • Renaming: Changing all entry points to a generic solution name.
  • Difficulty Stratification: They used a zero-shot code embedding model (SFR-Embedding-Code-400M_R) to score the similarity of each pair. Pairs that the model found "dissimilar" were labeled as "Hard," representing cases where the C and Rust implementations diverge structurally.
  • Identifier Ablation: To test if models were just matching variable names (like i, n, or temp), they created a version of the data where all user identifiers were replaced with positional placeholders (e.g., var1, var2).

Datasets

The primary contribution is CRUSTPairs, consisting of 1,857 C/Rust solution pairs.

As shown in the table below, CRUSTPairs fills a specific gap in the research landscape: it is the only dataset providing solution-level alignment specifically for the C-Rust language pair.

Comparison of cross-language code corpora. CRUSTPairs is unique in its C-Rust pairing at solution granularity.

The dataset is stratified by difficulty and source, ensuring that evaluations aren't skewed by easier problems from one specific archive.

Composition of the dataset by source and difficulty tier.

Key Insights and Ideas

  • Pretrained models are "Name-Dependent": In zero-shot settings, a simple lexical baseline (TF-IDF) actually beats a sophisticated pretrained model like UniXcoder. This happens because competitive programmers often reuse variable names across languages. The model isn't "understanding" the code; it's just doing fancy keyword matching.
  • Finetuning Learns Structure: Once finetuned on CRUSTPairs, models become remarkably robust. While the pretrained model's accuracy plummets when variable names are removed, the finetuned models retain 96–97% of their performance. This proves they have learned to map the underlying logic and control flow between C and Rust.
  • Names can be a Liability: On the "Hard" difficulty tier—where implementations diverge significantly—finetuned models actually perform better when identifiers are removed. In these complex cases, shared variable names are often coincidental and act as noise that distracts the model from the structural signal.

Key Results

The authors established retrieval baselines using Mean Reciprocal Rank (MRR). The jump from zero-shot to finetuned performance is dramatic:

Retrieval baselines on the test set. Finetuning (Full) provides a massive boost over zero-shot and lexical baselines.
  • Lexical TF-IDF: MRR 0.206
  • UniXcoder (Zero-shot): MRR 0.189 (Beaten by the simple lexical baseline!)
  • UniXcoder (Full Finetuning): MRR 0.792

The "Hard" tier remains a significant challenge, with zero-shot models scoring only 1.7x better than random chance, while full finetuning lifts that performance by 10x.

Limitations and Future Work

The authors are transparent about the dataset's boundaries:

  • Proxy for Systems Code: These are competitive programming solutions—short, single-file, and I/O-driven. They lack the complex memory allocation patterns, pointer arithmetic, and concurrency found in real-world "systems" code.
  • Functional Equivalence: While both solutions passed the same tests, they haven't been formally verified as identical.
  • Execution-based Verification: The authors suggest that future work should include execution-based checks to further validate the semantic equivalence of the pairs.
  • Pretraining Contamination: Since the source archives are public, there is a possibility that some of this code was seen by the models during their initial pretraining phase.

Summary updated

Abstract

C-to-Rust migration is an active agenda, but no dataset pairs the two languages at the level of whole solutions to a shared problem. We present CRUSTPairs, 1857 C/Rust implementation pairs drawn from competitive programming archives, normalised, deduplicated, and stratified into difficulty tiers by zero-shot cross-language embedding similarity, with perrecord provenance and licensing. Cross-language retrieval is hard on this data, and harder than a single baseline would suggest: TF-IDF over shared identifiers reaches MRR 0.21 and so beats a pretrained code model at 0.19, which on the hardest quarter of the data scores 1.7x chance. Finetuning reaches 0.79, and lifts that hardest quarter 10x. Renaming every user identifier settles what is being learned: the finetuned models keep 96-97% of their accuracy without them, while the pretrained model keeps 66\% - less than the lexical baseline does - and on structurally divergent pairs the finetuned models are better without identifiers than with them.
BibTeX
@misc{hejlek2026crustpairs,
  author = {Vojtech Hejlek and Leonardo Jesus Almeida and Alneu de Andrade Lopes},
  title = {{CRUSTPairs: A Cross-Language Dataset of Paired C and Rust Solutions}},
  year = {2026},
  eprint = {2611.11711},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11711},
}
Figure