Learning from LLM Agents: In-Context Generative Models for Text Casing in E-Commerce Ads

Yingxue Zhou, Tan Zhu, Tao Zeng, Zigeng Wang, Wei ShenEMNLP 2025

TL;DR

This paper tackles the challenge of ensuring correct sentence casing in e-commerce ads, a crucial aspect of readability and brand consistency. They introduce a two-stage approach: first, an LLM-powered "Chain-of-Actions" agent that accurately handles brand names, and second, a distilled, lightweight generative model that preserves this knowledge with significantly reduced latency and cost.

What Problem Does This Paper Address?

E-commerce platforms have strict content policies, including rules for sentence casing in ad text (headlines, calls to action). Proper casing involves capitalizing the first letter of a sentence while correctly capitalizing proper nouns, especially brand names. Existing methods, like NER-based models, struggle with domain-specific terms and context, while large language models (LLMs) like ChatGPT, though better at generalization, are expensive and slow, and can sometimes "hallucinate" incorrect information. This creates a need for a scalable, accurate, and efficient solution for automated ad compliance.

Methods

The authors propose a two-stage approach:

  1. LLM Chain-of-Actions (CoA) Agent: This stage uses an LLM (like GPT-40) to enforce casing policies. It works by:
  2. Brand Detection: Identifying potential brand names in the ad text.
  3. Brand Retrieval: Querying a "Brand Hub Library" to get the correct spelling and capitalization of detected brands.
  4. Prompt Construction: Incorporating the verified brand names into a prompt for the LLM.
  5. Sentence Rewrite: Using the LLM to generate the sentence-cased output, ensuring brand names are handled correctly. This CoA framework is designed to address LLM limitations with unseen entities.
  6. Distilled Generative Model: To overcome the latency and cost issues of LLMs, the knowledge from the CoA Agent is distilled into a lightweight sequence-to-sequence (Seq2Seq) model (based on T5).
  7. Pre-training: The model is first pre-trained on public news data to learn general sentence casing.
  8. Fine-tuning: It's then fine-tuned on ad-specific data, using the CoA Agent's outputs as labels.
  9. In-Context Decoding: A novel constrained decoding strategy is employed. This method restricts the model to generating only characters present in the original input, allowing only casing changes and preventing hallucinations.

Datasets

  • Public News Data: Approximately 6 million sentence pairs from the News Crawl WMT corpus were used for pre-training. This data was augmented with randomized casing to create input-target pairs.
  • Real Ads Data: Approximately 3,500 ads were collected, resulting in 7,700 text samples. These were annotated by the LLM CoA Agent to create 22,000 training pairs. An evaluation set of 1,000 samples was also created, with human annotators providing ground truth labels.

Key Insights and Ideas

  • Brand Names are a Key Challenge: Traditional NER models and even general LLMs struggle with correctly casing brand names, especially those that are rare or not in their training data. This is a critical failure point for ad compliance.
  • Chain-of-Actions for Robustness: Decomposing the task into distinct steps (detect, retrieve, rewrite) allows the LLM to leverage external knowledge (Brand Hub) and focus on specific sub-tasks, leading to more reliable brand name handling.
  • Distillation for Efficiency: The knowledge learned by a powerful but slow LLM can be effectively transferred to a much faster and cheaper model, making the solution practical for real-time applications.
  • In-Context Decoding Prevents Hallucinations: By constraining the generative model to only modify casing and not introduce new text, the risk of generating incorrect or fabricated content is significantly reduced.

Key Results

  • The proposed Distil-Ads-PN-ICD model (distilled model with in-context decoding) outperforms GPT-40 on real ad data in terms of accuracy (F1 score).
  • The distilled models achieve significantly reduced latency and cost compared to LLM-based approaches, making them suitable for large-scale deployment. For example, Distil-Ads-PN-ICD has a latency of 0.0042s and a projected cost of $0.000003 per sample, orders of magnitude better than CoA Agent or GPT-40.
  • The CoA Agent itself demonstrates superior performance over traditional NER baselines and even standalone LLMs on real ad data, highlighting the effectiveness of its structured approach.
  • Qualitative examples show that the CoA Agent and its distilled versions correctly handle brand names like "Babe" and acronyms like "FDA," while NER models fail.

Limitations and Future Work

  • Brand List Dependency: The system relies on an accurate and up-to-date brand name list. Incomplete or outdated lists could impact casing quality.
  • Lightweight Model Limitations: The distilled model might underperform on very rare linguistic patterns or highly ambiguous casing contexts compared to larger LLMs.
  • English-Centric Evaluation: The current evaluation is limited to English-language ads. Adapting the system to multilingual content or languages with different casing rules is a necessary next step.

Summary updated

BibTeX
@inproceedings{zhou2025learning,
  author = {Yingxue Zhou and Tan Zhu and Tao Zeng and Zigeng Wang and Wei Shen},
  title = {{Learning from LLM Agents: In-Context Generative Models for Text Casing in E-Commerce Ads}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure