C-SEO Bench: Does Conversational SEO Work?

Haritz Puerto, Martin Gubri, Tommaso Green, Seong Joon Oh, Sangdoo Yun

TL;DR

Conversational Search Engine Optimization (C-SEO)—the practice of tweaking web text so that AI search engines cite it first—is largely ineffective and often hurts citation rankings. Across a new multi-domain benchmark, traditional SEO (ranking high in the initial retriever so text appears early in the LLM’s context window) remains far more influential, while competitive C-SEO adoption quickly degrades into a congested, zero-sum game.

What Problem Does This Paper Address?

As search engines transition from ranked lists of links to Conversational Search Engines (CSEs) that generate synthesized answers with inline citations (e.g., Perplexity, ChatGPT Search, Google AI Overviews), content providers want to know how to optimize for visibility.

Early literature suggested that "white-hat" C-SEO—stylistic edits like making text more authoritative, adding statistics, or improving fluency—could boost an article’s presence in generated answers. However, prior studies evaluated narrow domains, relied on single-actor assumptions (assuming no competitors optimize their text), and evaluated visibility using raw word count rather than true citation priority. This paper investigates whether white-hat C-SEO genuinely helps content get cited earlier, how it performs across varied tasks and domains, and how competitive adoption affects performance.

Methods

The authors evaluate C-SEO by formalizing the objective: minimizing the citation rank ($\min \text{rank}(c_i)$) so that an LLM cites the target document earlier in its generated response.

The study tests ten white-hat transformation methods:

  • Eight stylistic transformations from prior work: Authoritative, Statistics, Citations, Fluency, Unique Words, Technical Terms, Simple Language, and Quotes.
  • Two newly proposed methods:
  • Content Improvement: A holistic rewrite combining all eight stylistic transformations into structured, polished text.
  • LLM Guidance: Prepending a markdown summary to the document, inspired by the llms.txt web standard.

Experiments were conducted using gpt-4o-mini to apply transformations, and evaluated on four frontier engines: gpt-4o-mini, claude-3-5-haiku, o3, and o4-mini. The authors also modeled competition by varying the adoption rate ($\alpha \in [0, 1]$), calculating the Area Under the Curve (AUC) across adoption levels to capture game-theoretic dynamics.

Datasets

The authors introduce C-SEO Bench, comprising over 1,900 queries and 16,300 documents across two primary tasks:

  • Product Recommendation (10 items retrieved, top 5 recommended):
  • Retail: 500 queries from the Amazon Shopping Queries dataset.
  • Video Games: 436 queries using Steam store tags and game descriptions.
  • Books: 249 genre/mood queries mapped to Google Books API entries.
  • Question Answering (5–8 candidate passages):
  • Web Questions: 300 queries from Natural Questions with web pages retrieved via Brave Search.
  • News: 294 events from the Multi-News dataset.
  • Debate: 142 controversial query snippets with supporting/opposing arguments.

Key Insights and Ideas

  • C-SEO is mostly ineffective—and often harmful: For most combinations of tasks, domains, and models, stylistic modifications did not produce statistically significant rank gains. In fact, aggressive strategies like adding arbitrary statistics frequently triggered significant rank demotions.
  • Context position (Traditional SEO) dominates generation: The order in which retrieved documents enter the LLM prompt (retrieval order) heavily dictates citation priority. Simply ranking first in the retrieval context yielded rank boosts up to an order of magnitude higher than any content modification. C-SEO cannot substitute for ranking well in traditional retrievers.
  • A congested, zero-sum game: As more competitors adopt C-SEO techniques, individual advantages dilute rapidly toward zero. Early adopters gain minimal margins that disappear as market adoption saturates.
  • Word count misled earlier findings: Prior claims of C-SEO success measured how many words the LLM used when discussing a source. However, more verbose coverage does not mean the LLM prioritized that document over competitors in citation order.

Key Results

  • Rare positive boosts: Out of 54 evaluated settings on gpt-4o-mini, only 3 showed statistically significant rank improvements: LLM Guidance on Retail (+0.36) and Video Games (+0.24), and Content Improvement on Retail (+0.18).
  • Zero significant wins on other models: On Claude 3.5 Haiku, o3, and o4-mini, no C-SEO method achieved statistically significant gains.
  • Frequent performance penalties: The Statistics method significantly decreased citation rankings in 19 out of 24 settings. In product recommendations on Haiku 3.5, 26 out of 30 settings had statistically significant negative effects.
  • Traditional SEO outclasses C-SEO: Placing a document first in the LLM context window (Best SEO) provided massive, statistically significant boosts across all domains (e.g., +2.77 positions in Retail, +1.89 in Games, +1.54 in Debate on gpt-4o-mini), completely dwarfing the best C-SEO gains (+0.36 max).

Limitations and Future Work

  • Exclusively white-hat: The study does not evaluate adversarial black-hat methods (such as prompt injections or jailbreaks aimed at actively suppressing competitors).
  • Context fit assumptions: All retrieved documents were short enough to fit inside the LLM context window; truncation or extreme long-context retrieval dynamics were not evaluated.
  • Linguistic and model scope: The benchmark is limited to English-language content and closed, commercial LLMs.
  • Future directions: Exploring potential synergies between SEO and C-SEO methods, testing open-weight models, and analyzing how intermediate document summarizers alter citation order.

Summary updated

Abstract

Large Language Models (LLMs) are transforming search engines into Conversational Search Engines (CSE). Consequently, Search Engine Optimization (SEO) is being shifted into Conversational Search Engine Optimization (C-SEO). We are beginning to see dedicated C-SEO methods for modifying web documents to increase their visibility in CSE responses. However, they are often tested only for a limited breadth of application domains; we do not know whether certain C-SEO methods would be effective for a broad range of domains. Moreover, existing evaluations consider only a single-actor scenario where only one web document adopts a C-SEO method; in reality, multiple players are likely to competitively adopt the cutting-edge C-SEO techniques, drawing an analogy from the dynamics we have seen in SEO. We present C-SEO Bench, the first benchmark designed to evaluate C-SEO methods across multiple tasks, domains, and number of actors. We consider two search tasks, question answering and product recommendation, with three domains each. We also formalize a new evaluation protocol with varying adoption rates among involved actors. Our experiments reveal that most current C-SEO methods are not only largely ineffective but also frequently have a negative impact on document ranking, which is opposite to what is expected. Instead, traditional SEO strategies, those aiming to improve the ranking of the source in the LLM context, are significantly more effective. We also observe that as we increase the number of C-SEO adopters, the overall gains decrease, depicting a congested and zero-sum nature of the problem.
BibTeX
@inproceedings{puerto2025c,
  author = {Haritz Puerto and Martin Gubri and Tommaso Green and Seong Joon Oh and Sangdoo Yun},
  title = {{C-SEO Bench: Does Conversational SEO Work?}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=oTeixD3oZO},
}
Figure