AudioTrace: File-Level Membership Detection for Audio Large Language Models

Guangzhi Sun, Xiao Zhan, Shuai Wang, Jose Such, Phil WoodlandarXiv 2026 · arXiv

TL;DR

AudioTrace is a novel framework for "file-level" membership inference in Audio Large Language Models (LLMs), allowing content owners to detect if their recordings were used for training without needing the original training labels. Alongside the method, the authors introduce AudioTraceBench, the first controlled benchmark for auditing privacy and copyright in the audio-to-text domain.

What Problem Does This Paper Address?

As Audio LLMs (like Qwen2.5-Omni or SALMONN) become more capable, they are increasingly trained on massive, publicly available audio datasets. This raises significant privacy and copyright concerns. Membership Inference Attacks (MIAs) are the standard tool for auditing this, but existing methods have a major blind spot: they usually require the exact "sample" (audio + transcript/question/answer) used in training.

In the real world, a model developer might use your podcast audio but pair it with their own privately generated captions or Q&A pairs. You have the audio file, but you don't have the "supervision" (the text). This paper tackles audio-file-level membership inference: determining if a specific recording was used for training, even when the associated training tasks and annotations are unknown.

Methods

The authors propose AudioTrace, a proxy-question-based approach. Instead of looking at the model's raw internal representations of the audio—which they show is ineffective—AudioTrace queries the model's behavior.

The process consists of three main stages:

  1. Proxy Question Generation: The target Audio LLM is prompted to generate its own questions about the candidate audio file. The intuition is that the model's learned associations will lead it to ask questions similar to those it saw during training.
  2. Response Generation & Metric Collection: The model then answers these self-generated questions. AudioTrace collects token-level statistics from these responses, such as Min-K% probability and predictive entropy.
  3. Audio-level Aggregation: The scores from multiple questions are aggregated (using a Top-K average) to produce a final membership score for the entire audio file.
The three stages of the AudioTrace membership inference method.

Datasets

The authors introduce AudioTraceBench, a controlled benchmark designed to avoid "temporal leakage" (where a model might have seen data during its initial pre-training).

  • Source: 1,285 unique audio recordings from YouTube videos published after January 1, 2026.
  • Models: Evaluated on three state-of-the-art Audio LLMs: Qwen2.5-Omni, SALMONN-2, and Step-Audio-2, all using pre-2026 checkpoints.
  • Composition: The dataset includes diverse categories like news, podcasts, and academic presentations to ensure a broad evaluation.
Distribution of audio categories in the AudioTraceBench dataset.

Key Insights and Ideas

  • Direct Methods Fail: Standard MIA techniques adapted from text and video (like measuring perplexity on audio-associated positions) perform near random guessing (AUROC ~0.50).
  • Probing Context Matters: Membership signals are much stronger in the model's responses to questions than in its internal audio representations.
  • Self-Generation is Key: Proxy questions generated by the target model itself are more effective than a fixed set of generic questions. This is because the model's "memory" of the training data influences the types of questions it generates.
  • Signal Strength vs. Exposure: The membership signal becomes clearer as the model is exposed to the data for more training epochs or when the model's adaptation capacity (e.g., LoRA rank) is increased.

Key Results

  • Performance Boost: AudioTrace achieves an AUROC of up to 0.60 across the three tested models, a significant improvement over the random-level performance of existing baselines.
  • Robustness: The method is robust to different scoring metrics (entropy vs. Min-K%) and works even in a black-box setting where only text outputs (not probabilities) are available, using response consistency as a signal.
  • Ablation Findings: Increasing the number of proxy questions improves performance, though gains start to diminish after about 10-30 questions.
AUROC improves as the number of proxy questions increases, stabilizing around 30 questions.

Limitations and Future Work

  • Computational Cost: AudioTrace is roughly 5x more computationally expensive than direct methods due to the multi-stage generation and scoring process.
  • SFT Focus: The study focuses on Supervised Fine-Tuning (SFT). Auditing membership in massive pre-training datasets remains an even larger, open challenge.
  • Signal Strength: While 0.60 AUROC is a step forward, it is still far from a "smoking gun." The authors emphasize that these scores should be treated as statistical evidence rather than definitive proof.
  • Future Directions: The authors suggest exploring adaptive query selection to further sharpen the membership signal, especially at low false-positive rates.

Summary updated

Abstract

Significant privacy and copyright concerns are raised by the use of publicly available audio recordings to train audio large language models (LLMs), yet effective auditing methods remain limited. Existing membership inference attacks typically assume knowledge of complete training examples, including their associated textual supervision. In practice, however, audio recordings may be paired with privately generated transcripts, captions, or question-answer pairs that are unavailable to owners of the recordings. This motivates audio-file-level membership inference, which determines whether a recording was used for training without assuming knowledge of its training tasks and annotations. This paper provides the first systematic investigation of this challenge and introduces AudioTraceBench, a controlled benchmark comprising 1,285 audio recordings with explicit fine-tuning membership labels. Evaluations across three audio LLMs show that existing text- and video-based membership inference methods adapted to audio perform close to random guessing. To address this challenge, AudioTrace is proposed, where audio-conditioned proxy questions are generated using the target model, and membership evidence is extracted from its responses and aggregated at the recording level. Consistent improvements over existing baselines are observed, with AUROC reaching 0.60 and increasing further under increased fine-tuning exposure or adaptation capacity. Further experiments were conducted to examine the factors influencing membership signals for audio-file-level detection. These findings establish audio-file-level membership inference as a distinct and practically important auditing challenge, with AudioTrace providing an effective initial approach to this challenge.
BibTeX
@misc{sun2026audiotrace,
  author = {Guangzhi Sun and Xiao Zhan and Shuai Wang and Jose Such and Phil Woodland},
  title = {{AudioTrace: File-Level Membership Detection for Audio Large Language Models}},
  year = {2026},
  eprint = {2611.21088},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21088},
}
Figure