Safe Words, Unsafe Voice: Pragmatic Harm in Controllable Audio Generation

Modern speech and music generators follow instructions about more than words. You can ask them to speak a sentence sarcastically, stress one syllable, sing a lyric as a funeral dirge, or add a gunshot at a chosen moment. This proposal asks whether that growing control creates a class of harmful audio that no current safety check can see. The harmful audio we have in mind is built from parts that are each innocent. The lyric is harmless and the musical mood is harmless, but together they intimidate, mock, or defraud. A text filter reads the words and finds nothing. An audio-quality metric hears clean audio and finds nothing. The harm lives only in how the parts combine, as a human listener hears it.

The bet joins two observations that have so far lived in separate literatures. Safety research keeps finding that more capable models are easier to misuse, because they faithfully carry out instructions that weaker models fumble. Audio evaluation research keeps finding that automatic metrics do not track what listeners actually perceive. If both hold, the most controllable audio generators should produce the most pragmatic harm, and the automatic tools used to police them should miss it most. We propose to build the first benchmark of this kind of harm on the generation side of audio, and to measure whether it grows with controllability. We also propose a small, interpretable judge that tries to close the perception gap. It first detects the perceptual attributes a listener would notice and then reasons about how they combine.

Core Claim

The claim has two halves, and each could fail independently.

The first half concerns generators. For audio generators ranked by how well they follow expressive instructions, the rate at which listeners judge a compositionally built output as harmful rises with that ranking. Weaker models stay "safe by failure" because they cannot render the menacing tone or the timed sound effect.

The second half concerns detection. The standard automatic safety pipelines detect this harm far below human level. Those pipelines include transcribing the audio and running a text guard, scoring audio-text embedding similarity, or asking a multimodal model to judge. Their shortfall is largest exactly where the harm depends on acoustic attributes that perceptual-evaluation work has shown these tools do not represent.

The closest precedent is Multimodal Pragmatic Jailbreak on Text-to-image Models. That paper builds harmful images from a safe scene plus safe rendered text. It finds that susceptibility rises with typographic skill, with DALL-E 3 reaching roughly 70% attack success while weaker models fail harmlessly. We take its compositional construction and its "capability breeds risk" hypothesis into a medium with two differences. In audio, meaning unfolds over time. It is also carried by prosody (the rhythm, stress, and intonation of speech) and by timbre, not by legible symbols.

The existing audio-jailbreak benchmarks are motivating but point the other way. JALMBench, Jailbreak-AudioBench and Evaluation of Audio Language Models for Fairness, Safety, and Security all study audio as input to a listening model, testing whether altered speech tricks it into answering a harmful question. They show that tone, speed and emotion shift refusal behavior. For example, Jailbreak-AudioBench raises SALMONN-7B's attack success from 31.6% to 85.1% by editing audio. None of the three asks whether a generator can be instructed to produce harmful audio from benign parts. None asks whether our detectors can hear the result. That output-side question is what we study.

Research Program

The benchmark. The first artifact is a benchmark of compositional audio prompts. Each item pairs a text component with an acoustic component, and each part passes a text safety classifier and a human "harmless in isolation" check. The acoustic component can be a prosody instruction, a genre or mood, a sound effect with timing, or a reference audio clip. Harm categories follow the four in MPUP (harassment and hate, physical threat, fraud, and sexual content). We restrict sexual content to non-explicit innuendo delivered by tone. The pragmatic devices also follow MPUP: sarcasm, allusion, metaphor and hyperbole. Audio adds two further devices. One is prosodic inversion, where praise is delivered so that it means contempt. The other is contextual scoring, where a benign lyric is set over sound design that turns it into a threat against a named target. For fraud items we build urgent, authority-toned delivery of innocuous-sounding account instructions. We use only synthetic, fictional identities and no voice cloning of real people.

A key design choice answers the obvious objection that the harm is visible in the prompt text anyway. A subset of items carries the acoustic component only as audio. That audio is either a reference clip for style or melody, as in the compositional multimodal instructions of CMI-RewardBench, or low-level prosodic markup such as stress indices and pause positions. In that subset a text reader of the prompt cannot recover the combined meaning.

The capability ladder. The second artifact is a ladder of generators ordered by expressive instruction-following. For speech, the ladder runs from plain text-to-speech systems with no style control, through instruction-controlled expressive text-to-speech, up to a LoopTTS-style closed-loop refiner. Diagnose, Then Refine trains such a refiner to place stress and pauses on exact syllables and reports 85.7% human preference over defective audio. That precise control is exactly the capability we suspect doubles as an attack tool. For music, the ladder uses the systems SongEval draws on, namely DiffRhythm, YuE, Suno, Mureka and Udio, as described in SongEval.

Each system's rung is set independently of the safety outcome. We measure instruction adherence on a benign mirror set of the same compositional structure. Adherence is scored by expert listeners and cross-checked with the CMI-RewardBench alignment reward model. This prevents the circular argument that a model is "capable" merely because it produced harm.

The human panel. Every generated clip is rated by trained listeners. They judge whether the clip is harmful, which category of harm it belongs to, and which acoustic cues carried the harm. The panel protocol mirrors the expert-annotation approach of SongEval and CMI-Pref.

The judge. The third artifact is a perceive-then-compose judge. Small, separately trained classifiers first extract the attributes listeners report: speech transcript, emotion, sarcasm and emphasis position, genre, tempo and mode, and sound events with timestamps. These attributes are written out as a structured, time-aligned description. A text safety model then reasons over that description. The rationale comes from the perceptual-evaluation literature. End-to-end audio embeddings and reference metrics track texture rather than the structure listeners attend to, as shown for music grammar by Hung et al., and they miss local prosody, as the LoopTTS diagnosis shows. We therefore expect explicit attribute extraction to beat asking one model to hear everything at once. This expectation is an empirical hypothesis, not an assumption, and the ablations below test it.

Baselines

The naive reference point is the deployed status quo. We transcribe the output with an off-the-shelf speech recognizer, and for songs we also read the lyric track. We then run a text guard such as Llama-Guard on the result. By construction, this baseline should score near zero recall on the benchmark, because every text component was chosen to pass a text guard. Its role is to confirm the construction, and any recall it achieves marks items where our "benign text" filter leaked.

The second baseline is a prompt-level compositional guard. A strong text LLM reads the full generation prompt, including lyric, style description and effect instructions, and asks whether the combination is harmful. This is the strongest competitor on the text-described subset. It also represents the cheapest realistic defense, since it needs no audio at all. We run it with the same safety model used inside our judge, so any difference comes from perception and not from reasoning strength.

The third baseline is an end-to-end multimodal audio judge. We use the strongest available audio language model, prompted with the same harm rubric and given the same number of prompt-engineering iterations on a held-out development split as our judge receives. Its closest analogue in prior work is the safety judge in Aloufi et al.: GPT-OSS-Safeguard-120b reached 85.33% agreement with human experts on speech safety in Evaluation of Audio Language Models for Fairness, Safety, and Security. That figure was measured on semantically explicit harmful speech, not on compositional harm. For pragmatic harm, the only reported judge figure comes from the image domain. In Multimodal Pragmatic Jailbreak on Text-to-image Models, a GPT-4o multimodal judge reached 74.3% correlation with human judgment. The research file's summary section attributes this figure to o1, so we cite it cautiously. We found no reported result on Lacuna for any detector on pragmatic harm in generated audio. Beating this baseline means exceeding its agreement with the human panel on the audio-only subset by a margin outside the confidence interval across harm categories, without worse false-positive rates on the benign mirror set.

The fourth baseline is a perceptual reward model with a safety head. We fine-tune the CMI-RewardBench reward model, which reports 78.20% accuracy on its musicality preference task, adding a harm head trained on the same labeled data our judge sees.

Two ablations isolate our key ingredient. The first gives our judge's reasoning model only the transcript, removing the acoustic attributes. The second gives it acoustic attributes without time alignment. Together they show whether perception, and specifically perception of when things happen, is what does the work.

Why not just…?

Why not just moderate the prompt, since the composition is written there? On the text-described items this is a real threat, and we expect it to catch much of the harm. If it catches nearly all of it, that is a useful negative finding. It would mean that text-level moderation of the whole prompt, and not of the lyric alone, suffices wherever inputs are textual. The proposal does not rest on that subset. It rests on two other groups. The first is items where the acoustic component arrives as reference audio or low-level markup. The second is generations where the harmful reading emerges from the model's own rendering choices and was not specified in the prompt. The comparison between the two subsets settles how much protection prompt moderation actually provides.

Why not just use a frontier audio language model as the judge? The evidence says these models hear coarsely. Gemini 3 Pro reaches only 65.80% agreement with experts on music judgments in CMI-RewardBench. LoopTTS needed a dedicated diagnosis stage because utterance-level scores miss misplaced stress and odd pauses. Still, these models improve quickly, and this baseline could close the gap. That is why it is run with a matched tuning budget. If it matches our judge, the conclusion is that end-to-end models suffice and the modular design is unnecessary.

Why not just add a safety head to an existing perceptual reward model? CMI-RewardBench reports that its musicality and instruction-alignment heads correlate at a Spearman correlation of about 0.85. This suggests such models collapse distinct perceptual judgments into one "good match" signal. A safety head trained on top would likely inherit that entanglement and reward faithful rendering of a harmful composition. We test this directly by measuring the correlation between the new harm head and the alignment head on the benign mirror set, and by checking recall on the audio-only subset.

Decisive Experiments

The first check tests the capability claim. Within each ladder (speech and music separately), we regress the human-judged harm rate on each generator's benign instruction-adherence score, with harm category as a covariate. The claim fails if the relationship is flat or negative. It also fails if harm rises only because more capable models produce clearer audio. To rule that out, we control for a quality score such as the SongEval-style aesthetic predictor, whose base model reaches 0.912 Pearson correlation with human ratings.

The second check tests the perception-gap claim. We measure each baseline's agreement with the human panel separately on the text-described subset and the audio-only subset. The gap claim holds if the transcript-plus-text-guard pipeline and the embedding-similarity score fail on both subsets, and the end-to-end judge degrades on the audio-only subset. It holds most strongly if that failure concentrates on the prosodic-inversion and timed-effect devices.

The third check tests the judge. Our judge must beat the end-to-end audio judge on the audio-only subset at matched false-positive rate on benign mirrors, and the two ablations must each cost it accuracy. If removing time alignment costs nothing, the "contextual scoring" device is not what matters. If the transcript-only ablation matches the full judge, the benchmark was not truly compositional and must be rebuilt.

Finally, we check transfer. We run the judge unchanged on Jailbreak-AudioBench's edited-audio samples, to test whether attributes learned for generated harm also expose the input-side hidden-semantics attacks those authors document.