TL;DR
CRUSTPairs is the first dataset of aligned C and Rust solutions at the whole-program level, designed to benchmark cross-language retrieval and similarity. The authors demonstrate that while pretrained code models often rely on superficial variable name matching, finetuning on this dataset allows models to learn deep structural mappings that remain robust even when all identifiers are removed.
What Problem Does This Paper Address?
Memory safety vulnerabilities are a persistent plague in systems programming, with Microsoft and Android attributing roughly 70% of their CVEs to this class of defect. The industry is aggressively moving toward Rust to mitigate these risks, but migrating legacy C codebases is a monumental task.
While automated translation tools are emerging, evaluating them is tricky. Current benchmarks focus on whether the output compiles and passes tests. However, legacy code often lacks comprehensive test suites. This creates a need for a different kind of evaluation: semantic similarity. We need to be able to judge if a Rust program is a faithful rendering of a C source without relying solely on execution. Before CRUSTPairs, no dataset existed that paired these two languages at the level of complete, functional solutions.
Methods
The authors constructed CRUSTPairs by pulling accepted solutions for the same competitive programming problems from two major archives (CodeNet and xCodeEval). To ensure the models learn meaningful patterns rather than just "cheating" by looking at comments or specific function names, they applied a rigorous preprocessing pipeline:
- Normalizing: Stripping comments, removing unused code, and applying standard formatters.
- Renaming: Changing all entry points to a generic
solutionname. - Difficulty Stratification: They used a zero-shot code embedding model (SFR-Embedding-Code-400M_R) to score the similarity of each pair. Pairs that the model found "dissimilar" were labeled as "Hard," representing cases where the C and Rust implementations diverge structurally.
- Identifier Ablation: To test if models were just matching variable names (like
i,n, ortemp), they created a version of the data where all user identifiers were replaced with positional placeholders (e.g.,var1,var2).
Datasets
The primary contribution is CRUSTPairs, consisting of 1,857 C/Rust solution pairs.
As shown in the table below, CRUSTPairs fills a specific gap in the research landscape: it is the only dataset providing solution-level alignment specifically for the C-Rust language pair.
The dataset is stratified by difficulty and source, ensuring that evaluations aren't skewed by easier problems from one specific archive.
Key Insights and Ideas
- Pretrained models are "Name-Dependent": In zero-shot settings, a simple lexical baseline (TF-IDF) actually beats a sophisticated pretrained model like UniXcoder. This happens because competitive programmers often reuse variable names across languages. The model isn't "understanding" the code; it's just doing fancy keyword matching.
- Finetuning Learns Structure: Once finetuned on CRUSTPairs, models become remarkably robust. While the pretrained model's accuracy plummets when variable names are removed, the finetuned models retain 96–97% of their performance. This proves they have learned to map the underlying logic and control flow between C and Rust.
- Names can be a Liability: On the "Hard" difficulty tier—where implementations diverge significantly—finetuned models actually perform better when identifiers are removed. In these complex cases, shared variable names are often coincidental and act as noise that distracts the model from the structural signal.
Key Results
The authors established retrieval baselines using Mean Reciprocal Rank (MRR). The jump from zero-shot to finetuned performance is dramatic:
- Lexical TF-IDF: MRR 0.206
- UniXcoder (Zero-shot): MRR 0.189 (Beaten by the simple lexical baseline!)
- UniXcoder (Full Finetuning): MRR 0.792
The "Hard" tier remains a significant challenge, with zero-shot models scoring only 1.7x better than random chance, while full finetuning lifts that performance by 10x.
Limitations and Future Work
The authors are transparent about the dataset's boundaries:
- Proxy for Systems Code: These are competitive programming solutions—short, single-file, and I/O-driven. They lack the complex memory allocation patterns, pointer arithmetic, and concurrency found in real-world "systems" code.
- Functional Equivalence: While both solutions passed the same tests, they haven't been formally verified as identical.
- Execution-based Verification: The authors suggest that future work should include execution-based checks to further validate the semantic equivalence of the pairs.
- Pretraining Contamination: Since the source archives are public, there is a possibility that some of this code was seen by the models during their initial pretraining phase.
Summary updated