Ranking-Incentivized Quality Preserving Content Modification

Gregory Goren, Oren Kurland, Moshe Tennenholtz, Fiana RaiberSIGIR 2020 · DOI · OpenReview

Abstract

The Web is a canonical example of a competitive retrieval setting where many documents' authors consistently modify their documents to promote them in rankings. We present an automatic method for quality-preserving modification of document content --- i.e., maintaining content quality --- so that the document is ranked higher for a query by a non-disclosed ranking function whose rankings can be observed. The method replaces a passage in the document with some other passage. To select the two passages, we use a learning-to-rank approach with a bi-objective optimization criterion: rank promotion and content-quality maintenance. We used the approach as a bot in content-based ranking competitions. Analysis of the competitions demonstrates the merits of our approach with respect to human content modifications in terms of rank promotion, content-quality maintenance and relevance.

Introduction

The Web search ecosystem is a dynamic, adversarial environment where document authors act as "players" competing for visibility. High rankings translate directly into utility, such as clicks and user engagement. While much of the research in adversarial Web search focuses on detecting "black hat" spamming techniques, there is a significant gap in understanding "white hat" Search Engine Optimization (SEO)—legitimate modifications intended to improve a document's representation relative to a query without compromising quality.

This paper addresses two primary challenges in automated white-hat SEO:

  1. Rank Promotion: Identifying modifications that will improve a document's position within a non-disclosed, "black box" ranking algorithm.
  2. Quality Preservation: Automating these changes while maintaining the document's coherence, readability, and relevance.

Methodology

The proposed method utilizes an automatic passage replacement strategy. The system identifies a "source" passage within a document and replaces it with a "target" passage selected from a candidate pool of passages from other highly-ranked documents.

Learning-to-Rank (LTR) Framework

The core of the system is a passage-pair ranker that evaluates potential (source_passage, target_passage) swaps using a bi-objective optimization criterion:

  • Rank Promotion ($r$): A prediction of the improvement in the document's ranking position.
  • Content-Quality Maintenance ($c$): A measure of how well the new passage fits the local context.

These objectives are combined using a smoothed harmonic mean, allowing for a tunable balance between aggressive rank optimization and conservative quality preservation.

Feature Engineering

The model leverages two main feature sets:

  • Rank-Promotion Features: These include query term frequency and the similarity of the candidate passage to documents that are currently or historically highly ranked for the target query. This "mimicry" strategy allows the bot to adopt characteristics favored by the search engine.
  • Coherence Features: Word2Vec embeddings are used to measure the semantic similarity between the candidate passage and the original document's surrounding context (preceding and following passages).

Experimental Setup

The bot was evaluated in a live competitive retrieval setting. Human students were tasked with writing and modifying documents to rank for 15 TREC queries (2009-2012). The bot competed directly against these students over multiple rounds. The source corpus for document modification was the ClueWeb09 Category B collection. Training data was derived from a previous ranking competition (Raifer et al., 2017).

Results and Discussion

The bot demonstrated superior performance compared to human participants across several key metrics.

Performance metrics of the bot vs. human students across competition rounds.

As shown in the combined metrics above, the bot achieved an average rank of 2.667 in the second round, significantly outperforming the human students' average of 3.400. Notably, the bot was the only participant to achieve positive "scaled promotion" values, indicating consistent upward movement in the rankings. While human students often struggled to improve their rankings in subsequent rounds—sometimes even experiencing demotions—the bot's performance remained more stable.

Furthermore, the bot maintained high standards for content quality. Despite being an automated system, it achieved a relevance rate of 86.6% and a quality score of 0.933, often surpassing the quality of human-modified documents. This suggests that rank promotion and quality maintenance are not necessarily mutually exclusive.

Scaled promotion achieved by the bot and students relative to the document's initial rank.

Analysis of promotion relative to initial rank (above) reveals that the bot consistently outperformed students regardless of the starting position. While students' ability to promote documents fluctuated significantly based on the initial rank, the bot's systematic approach provided a more reliable path to rank improvement.

Conclusion

The success of the automated bot highlights that observing the "winners" in a search result provides a strong signal for optimization, even without knowledge of the underlying ranking function. However, the study also noted limitations, such as a slight decay in performance over multiple iterative rounds and the ethical/copyright implications of using existing documents as a candidate pool. Future work may involve using generative models (e.g., LLMs) to create original, high-quality content for replacement, further refining the balance between visibility and document integrity.

Summary updated

Abstract

The Web is a canonical example of a competitive retrieval setting where many documents' authors consistently modify their documents to promote them in rankings. We present an automatic method for quality-preserving modification of document content --- i.e., maintaining content quality --- so that the document is ranked higher for a query by a non-disclosed ranking function whose rankings can be observed. The method replaces a passage in the document with some other passage. To select the two passages, we use a learning-to-rank approach with a bi-objective optimization criterion: rank promotion and content-quality maintenance. We used the approach as a bot in content-based ranking competitions. Analysis of the competitions demonstrates the merits of our approach with respect to human content modifications in terms of rank promotion, content-quality maintenance and relevance.
BibTeX
@inproceedings{goren2020ranking,
  author = {Gregory Goren and Oren Kurland and Moshe Tennenholtz and Fiana Raiber},
  title = {{Ranking-Incentivized Quality Preserving Content Modification}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2020},
  doi = {10.1145/3397271.3401058},
}

Loading related content...

Figure