asLLR: LLM based Leads Ranking in Auto Sales

Yin Sun, Yiwen Liu, Junjie Song, Chenyu Zhang, Xinyuan Zhang, Lingjie Liu, Siqi Chen, Yuji CaoarXiv 2025 · DOI · arXiv

Abstract

In the automotive industry, identifying high-potential sales leads from a vast pool of inquiries is critical for operational efficiency. Traditional lead scoring models primarily utilize structured tabular data, often overlooking the rich, unstructured information contained in customer-sales interactions. This paper introduces asLLR (LLM-based Leads Ranking), a framework that leverages Large Language Models (LLMs) to integrate both tabular features and conversational text. By employing a dual-loss training strategy—combining standard Click-Through Rate (CTR) prediction with a natural language Question Answering (QA) task—asLLR significantly improves ranking performance. Real-world deployment at a major New Energy Vehicle (NEV) brand resulted in a 9.5% increase in sales volume and a substantial AUC gain of 0.0231 over traditional baselines.

Introduction

Lead scoring is a fundamental challenge in auto sales. Sales teams are frequently overwhelmed by inquiries, necessitating a system that can accurately prioritize leads likely to convert. Traditional Click-Through Rate (CTR) models excel at processing structured "tabular" data, such as call frequency or test drive counts. However, they struggle to capture the qualitative nuances of human interaction—excitement, specific concerns, or intent—found in CRM text logs. The asLLR framework addresses this by treating lead scoring as a "Learning-to-Rank" problem that digests both rigid numerical data and messy natural language.

Methodology: The asLLR Framework

The asLLR model utilizes a decoder-only architecture, specifically the Qwen1.5 family, fine-tuned via Low-Rank Adaptation (LoRA) for computational efficiency.

The asLLR architecture integrates tabular features and conversational logs through a dual-loss training objective.

Dual-Loss Training

The core innovation of asLLR is its dual-loss objective:

  1. CTR Loss: A standard point-wise ranking loss used to predict binary purchase outcomes.
  2. QA Loss: The model is tasked with answering: "Based on this info, will the customer buy the car?" This objective forces the LLM to leverage its pre-trained linguistic reasoning, acting as a regularizer that prevents overfitting on numerical features.

Text Processing and Feature Extraction

To handle long and noisy Automatic Speech Recognition (ASR) transcripts, a summarization module reduces token length by approximately 60%, enhancing the signal-to-noise ratio. Furthermore, asLLR can function as a feature extractor, generating 128-dimensional embeddings from text logs to augment traditional models like DeepFM.

Dataset and Data Integrity

The study utilizes a proprietary dataset from a prominent NEV brand, comprising 300,000 training and 40,000 testing samples. Features include 31 tabular behavioral statistics and unstructured communication records.

Temporal partitioning is employed to prevent feature and label leakage, ensuring the model does not see future information during training.

To ensure model integrity, the researchers implemented a strict temporal split. As illustrated in the figure above, improper partitioning can lead to "data leakage" where future labels or features inadvertently inform past predictions. By training on historical data and testing on subsequent time periods, the evaluation reflects real-world predictive performance.

Key Results and Insights

Experimental results demonstrate the superiority of the LLM-based approach:

  • Performance Gains: asLLR achieved an AUC of 0.8127, outperforming traditional models like DeepFM and DCN by 0.0231.
  • Scaling and Regularization: Performance scaled with model size (from 1.8B to 14B parameters). Notably, the inclusion of QA loss was vital; without it, the model overfitted rapidly to the tabular data.
  • Real-World Impact: A five-month online A/B test showed a 9.5% increase in actual sales volume for the group prioritized by asLLR.

Limitations and Future Work

Despite its success, asLLR faces challenges regarding ASR quality; transcription errors in noisy environments can degrade performance. Additionally, the team noted a "U-shaped" dropout performance curve and potential catastrophic forgetting of general knowledge during task-specific fine-tuning. Future research will focus on advanced knowledge compression and improved ASR integration to handle longer customer journeys.

Summary updated

BibTeX
@misc{sun2025asllr,
  author = {Yin Sun and Yiwen Liu and Junjie Song and Chenyu Zhang and Xinyuan Zhang and Lingjie Liu and Siqi Chen and Yuji Cao},
  title = {{asLLR: LLM based Leads Ranking in Auto Sales}},
  year = {2025},
  eprint = {2510.21713},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.21713},
}
Figure