From Text to Sound: A Preliminary Study on Retrieving Sound Effects to Radio Stories

Songwei Ge, Curtis Xuan, Ruihua Song, Chao Zou, Wei Liu, Jin Zhou

Abstract

Adding sound effects (SFX) to radio stories is a labor-intensive process that currently relies on human intuition. This paper proposes a retrieval-based framework that uses a semantic inference model to determine not just which sound to play, but if a sound should be played at all. By addressing linguistic ambiguities such as metaphors, negation, and future intent, the proposed system significantly reduces false positives, paving the way for more immersive and automated audio storytelling.

Introduction

While text-to-speech (TTS) technology has made massive strides in generating human-like narration, radio stories still feel "flat" without background atmosphere and sound effects. The current state of the art in automated audio production often relies on simple keyword-based retrieval: if the text contains the word "forest," the system plays a forest sound. However, this naive approach fails to account for the complexity of natural language, leading to jarring audio experiences where sounds are triggered out of context.

The Challenge of Linguistic Ambiguity

Language is inherently messy, and a keyword-based system fails in several critical scenarios. The researchers identified that over 40% of triggers suggested by simple retrieval are "false positives" that would ruin a listener's immersion. Key failure modes include:

  1. Metaphors and Titles: A phrase like "The king of the forest" does not necessarily mean the scene takes place in a forest.
  2. Future Intent: "He plans to go to the forest" implies the character is not yet there; playing ambient bird sounds immediately would be premature.
  3. Negation: A sentence stating "There was no wind" should explicitly not trigger a wind sound effect.
Figure 2: Examples of sentences where identical keywords (in bold) either should or should not trigger a sound effect based on semantic context.

Proposed Framework

To solve these issues, the researchers developed a pipeline that moves beyond simple tag matching. The core contribution is a Semantic Inference Model that acts as a gatekeeper for sound triggers.

Figure 1: The proposed system architecture, illustrating the flow from story annotation to semantic inference and final high-quality radio story synthesis.

As shown in Figure 1, the pipeline consists of several stages:

  1. Candidate Trigger Identification: A sound-tag database (built from sources like Freesound and Adobe Audition) identifies keywords in the story text that could potentially be sounds.
  2. Semantic Inference: This model evaluates the context of the candidate trigger using feature-based classifiers (SVM and XGBoost).
  3. Synthesis: The validated triggers are combined with TTS narration and background music to produce the final radio story.

Feature Engineering and Heuristics

Because large-scale annotated datasets for this specific task are scarce, the authors utilized a feature-engineering approach rather than end-to-end deep learning. They extracted three primary types of features:

  • Special Words: Counts of action words (e.g., knocking, crying), weather terms, negative words (e.g., not, none), and time-based words.
  • Part of Speech (POS): Identifying if a trigger is a noun (likely a scene) or an adjective (likely descriptive).
  • Syntactic Features: Using dependency parsing to determine if a trigger is the subject or object of a verb, which often signals an active sound-producing event.

Additionally, manual heuristic rules were implemented to handle edge cases, such as ignoring triggers found inside quotations or following similes (e.g., "as if" or "like").

Key Results and Insights

The semantic inference model significantly outperformed baseline keyword retrieval. Using an SVM classifier, the system achieved a precision of ~70.2% and a recall of ~77.2%. When manual heuristic rules were added, precision jumped to 75.4%, though recall dropped to 63.3%. In the context of radio production, the authors argue that high precision is more important than high recall; it is far better to miss a subtle sound effect than to play a jarringly incorrect one.

Two major insights emerged from the data:

  • Action is Everything: "Action words" were the most critical feature. If a sentence contains a verb related to the sound (e.g., "the river roars"), the probability that the sound is appropriate increases dramatically.
  • The "Now" Paradox: Surprisingly, including time-based words like "now" actually worsened the model's predictions. These words proved difficult to generalize across different story contexts, leading to distribution discrepancies in the training data.

Conclusion

This preliminary study demonstrates that semantic context is vital for automated sound effect retrieval. While the current model is constrained to sentence-level analysis, it provides a robust foundation for future work. Next steps include exploring larger context windows to allow sounds to persist across paragraphs and applying neural networks to capture more nuanced semantic relationships. Furthermore, the complex task of automated audio mixing—determining volume levels and fade durations—remains an open challenge for the field.

Summary updated

Abstract

Sound effects play an essential role in producing high-quality radio stories but require enormous labor cost to add. In this paper, we address the problem of automatically adding sound effects to radio stories with a retrieval-based model. However, directly implementing a tag-based retrieval model leads to high false positives due to the ambiguity of story contents. To solve this problem, we introduce a retrieval-based framework hybridized with a semantic inference model which helps to achieve robust retrieval results. Our model relies on fine-designed features extracted from the context of candidate triggers. We collect two story dubbing datasets through crowdsourcing to analyze the setting of adding sound effects and to train and test our proposed methods. We further discuss the importance of each feature and introduce several heuristic rules for the trade-off between precision and recall. Together with the text-to-speech technology, our results reveal a promising automatic pipeline on producing high-quality radio stories.
BibTeX
@inproceedings{ge2019from,
  author = {Songwei Ge and Curtis Xuan and Ruihua Song and Chao Zou and Wei Liu and Jin Zhou},
  title = {{From Text to Sound: A Preliminary Study on Retrieving Sound Effects to Radio Stories}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2019},
  doi = {10.1145/3331184.3331274},
}
Figure