TL;DR
This paper evaluates how effectively open-weight Large Language Models (LLMs) can detect and diagnose critical vulnerabilities in Solana cross-chain bridges. The researchers found a massive gap between "raising an alarm" and "correctly diagnosing a bug": while models frequently flagged code as vulnerable, they only correctly identified the specific underlying flaw in 24% of source-code cases, often relying more on code comments than executable logic.
What Problem Does This Paper Address?
Cross-chain bridges are the "high-security vaults" of the crypto world, locking up billions in collateral to issue wrapped tokens on other chains. However, they are notoriously difficult to secure. On the Solana blockchain, the runtime environment handles basic memory safety, but it leaves "application-level" validation—like checking if an account is actually signed by an admin or if a token belongs to the right "mint"—entirely to the developer.
Missing these checks has led to catastrophic losses, such as the $320M Wormhole hack and the $52M Cashio incident. While LLMs are increasingly used for security audits, we don't yet know if they truly "understand" these subtle Solana-specific logic flaws or if they are just guessing based on patterns. This paper seeks to quantify that understanding by testing models on both raw source code and transaction execution records.
Methods
The authors built a custom "lock-and-mint" bridge in native Rust for Solana and created four vulnerable variants based on real-world exploits and audit classes:
- A1 (Mint Confusion): Accepting worthless tokens as collateral (modeled after Cashio).
- A2 (Sysvar Spoofing): Bypassing signature checks by providing a fake instructions account (modeled after Wormhole).
- B1 (Missing Signer): Allowing admin changes without a valid signature.
- B2 (Arbitrary CPI): Letting an attacker substitute a malicious program during a cross-program invocation.
They evaluated six configurations of five open-weight models (including DeepSeek-V3.2, Llama-3.3-70B, and gpt-oss-120b) across two tracks:
- Source-Code Auditor: Reviewing Rust code to find bugs.
- Transaction-Record Monitor: Analyzing logs and balance changes from a completed session to detect an attack.
To ensure rigorous results, they used a "blinded" grading protocol where two independent LLMs (Claude and Gemini) graded the responses against a strict rubric to distinguish between a lucky "yes, it's buggy" guess and a correct technical diagnosis.
Datasets
The study introduced a new benchmark dataset consisting of:
- The Bridge Implementation: A native Rust Solana bridge with 4 vulnerable variants and 1 secure baseline.
- Transaction Traces: Captured records of successful attacks, benign operations, and blocked attempts.
- Model Responses: A massive set of 16,800 responses from the evaluated LLMs.
- Graded Labels: 7,200 human-verified, LLM-assisted grades for the "vulnerable" inputs.
Key Insights and Ideas
- Alarms are Cheap, Diagnosis is Expensive: Most models are "alarm-happy." In many configurations, they flagged 93-100% of secure code as vulnerable. This makes them "noisy" tools that could overwhelm human auditors with false positives.
- The "Comment" Crutch: When original code comments were restored, the models' ability to diagnose bugs jumped from 24% to 60%. This suggests models are often "reading the map, not the terrain"—they find the bug because a comment describes a check that happens to be missing, rather than inferring the need for that check from the logic itself.
- Transaction Context Matters: Models performed very differently depending on the input. For example, the "Missing Signer" (B1) bug was rarely caught in source code but was identified in nearly 100% of transaction records because the lack of a signature is glaringly obvious in an execution trace.
- Thinking Helps (Sometimes): Enabling "thinking" (Chain-of-Thought) in DeepSeek-V3.2 significantly improved its performance in transaction-level detection (from 4% to 91% across three scenarios), though it had negligible impact on source-code auditing.
Key Results
- Source-Code Diagnosis: Without comments, models correctly diagnosed only 24% of vulnerable inputs.
- False Positive Rate: Five out of six model configurations flagged nearly every program—even the secure ones—as buggy, leading to an accuracy of only 50-52% on a balanced dataset.
- Top Performer: gpt-oss-120b showed the best discrimination, withholding alarms on about one-third of secure runs while achieving the highest diagnosis count (218 out of 400).
- Transaction Monitoring: While high for the "Missing Signer" scenario, diagnosis averaged only 6% across the other three transaction-based attack scenarios for most models.
Limitations and Future Work
The study focused on native Rust implementations rather than the popular Anchor framework, which automates many of these checks. Future work could explore if LLMs are better at auditing Anchor-based contracts. Additionally, the study used a limited set of four vulnerability types; a broader "zoo" of exploits would provide a more comprehensive picture.
The authors also suggest that future research should investigate "adversarial comments"—outdated or misleading documentation—to see if they can trick LLMs into giving a clean bill of health to buggy code.
Summary updated