Abstract
In the design of customer relationship management (CRM) systems, accurately identifying customer types and offering personalized services are key to enhancing customer satisfaction and loyalty. However, this process faces the challenge of discerning customer voices and intentions, and general pre-trained automatic speech recognition (ASR) models make it difficult to effectively address industry-specific speech recognition tasks. To address this issue, we innovatively proposed a solution for fine-tuning industry-specific ASR models, which significantly improved the performance of the fine-tuned ASR models in industry applications. Experimental results show that our method substantially improves the crucial auxiliary role of the ASR model in industry CRM systems, and this approach has also been adopted in actual industrial applications.
The Industrial Context
Modern CRM systems are increasingly incorporating voice-to-text features to help sales personnel record customer preferences and intentions in real-time. This process, illustrated in the pipeline below, transforms raw speech into actionable customer portrait labels that drive decision support and personalized service.
However, general-purpose ASR models (like OpenAI’s Whisper) struggle in specific industrial contexts due to three primary challenges:
- Data Scarcity: Obtaining large volumes of accurately labeled, industry-specific audio is expensive and time-consuming.
- Domain Jargon: Real-world sales data is filled with proprietary brand names, colloquialisms, and regional accents.
- Multilingual Mix: In global industries like luxury goods, speech often mixes languages (e.g., Chinese sentences with English brand names), making standard error metrics like Word Error Rate (WER) insufficient.
The Weak Supervision Pipeline
To bridge the data gap without massive manual labeling, we propose a "Weak Supervision" framework that leverages Large Language Models (LLMs) and Text-to-Speech (TTS) technologies to generate high-quality training data.
Methodology
- Data Expansion (LLM + TTS): We utilize DeepSeek V2 to generate synthetic text labels based on a small seed of real data and a curated list of 859 industry-specific keywords. These labels are fed into ChatTTS to create synthetic audio. To ensure robustness, we inject "interference" such as accents and dialects into the TTS process.
- Data Filtering: To maintain dataset quality, a pre-trained Whisper model transcribes the generated audio. If the transcription differs from the original synthetic label by a Character Error Rate (CER) greater than 0.15, the data point is discarded.
- Fine-tuning: The filtered synthetic dataset (approximately 20,000 pairs) is used to fine-tune Whisper models using LoRA (Low-Rank Adaptation), allowing for efficient training with lower computational overhead.
- Integrated Error Rate (IER): We developed a new metric that combines WER, CER, and a specific weight for keywords to provide a holistic view of model performance in mixed-language, jargon-heavy environments.
Key Results and Insights
Our experimental results demonstrate that the "Synthetic Flywheel"—using a small seed of real data to hallucinate a massive training set—is highly effective.
- Significant Accuracy Gains: Fine-tuned models showed an average performance improvement of 51.5% over native baselines, with the highest recorded improvement reaching 63%.
- Model Selection: Surprisingly, Whisper-large-v2 outperformed the newer v3 after fine-tuning. The v3 model exhibited "abnormal repetition" of Chinese characters, suggesting that larger models may be more prone to learning biases when fine-tuned on synthetic data.
- The Necessity of Filtering: The verification step using a pre-trained model is critical; without it, noise in the synthetic data can degrade rather than improve performance.
Conclusion
This framework has already been deployed in industrial CRM applications, receiving positive feedback for its ability to accurately capture customer preferences. While challenges remain regarding semantic drifts in LLM generation and TTS artifacts, our approach provides a scalable path for adapting general ASR models to specialized industrial domains.
Summary updated