CHANGE-IT @ EVALITA 2020: Change Headlines, Adapt News, GEnerate (short paper)

Lorenzo De Mattei, Michele Cafagna, Felice Dell'Orletta, Malvina Nissim, Albert GattEVALITA 2020 · OpenReview

Introduction

While Natural Language Generation (NLG) has seen explosive growth in English, other languages—including Italian—have historically had fewer benchmarks and shared tasks to stimulate research. The CHANGE-IT task, introduced for the EVALITA 2020 campaign, addresses this gap by focusing on "headline translation" between newspapers at opposite ends of the political spectrum. Beyond the language gap, the authors tackle a fundamental hurdle in NLG: evaluation.

Evaluating style transfer (e.g., changing a "left-leaning" headline to a "right-leaning" one) is notoriously difficult. Human evaluation is expensive and often inconsistent because people struggle to identify subtle stylistic nuances. To solve this, the authors designed the CHANGE-IT task, providing both the data and a robust automatic metric to determine if a generated headline successfully adopts a new style without losing its original meaning.

Methodology: Style Transfer as Extreme Summarization

The authors frame style transfer as a "headline translation" problem. To provide a starting point for researchers, they implemented a baseline system using pointer-generator networks.

What makes their approach interesting is that they treat style transfer as an extreme summarization task. Instead of a direct text-to-text rewrite, the baseline models are trained as article-to-headline generators. By training one model on la Repubblica (left) and another on Il Giornale (right), they can "transfer" style by feeding an article from one paper into the generator trained on the other.

The Evaluation Framework

The core technical contribution is a three-part automatic evaluation strategy:

  1. Main Classifier: A BERT-based model trained to identify which newspaper a headline came from. A style transfer is "successful" if it manages to flip this classifier's prediction.
  2. HH (Headline-Headline) Classifier: A "sanity check" that ensures the generated headline still describes the same content as the original.
  3. AH (Article-Headline) Classifier: Another check to ensure the new headline is still compatible with the source article.
Performance metrics for the three classifiers used in the evaluation framework.

These are combined into an Overall Compliancy score, representing the proportion of generated headlines that pass all three tests.

Dataset and Alignment

The authors released a substantial new dataset for the Italian NLP community consisting of approximately 152,000 article-headline pairs from la Repubblica (left-leaning) and Il Giornale (right-leaning).

Overall data splits and training/test set configurations.

To facilitate training, they used tf-idf cosine similarity to find "aligned" news (articles from both papers covering the same event). As shown in the figure above, the data is categorized into "Strongly Aligned" (A1, A2, A3), "Weakly Aligned," and "Not Aligned" (R). This structure allows researchers to train on a mix of parallel and non-parallel data.

Key Results and Insights

The baseline system demonstrated that the task is non-trivial. The Main Classifier was more easily "fooled" when transferring style from la Repubblica to Il Giornale (.799) than vice versa (.435). However, the AH classifier (article-headline match) remained high (~.87), suggesting the models were good at staying relevant to the article.

The average Overall Compliancy score was .345. While this seems low, it serves as a realistic floor for future models to beat, highlighting the difficulty of simultaneously changing style and preserving exact meaning. The authors note that humans often find it hard to distinguish these specific newspaper styles; by using a BERT-based classifier as the "judge," they provide a more sensitive and reproducible metric for stylistic shifts than typical human surveys.

Conclusion and Future Work

The primary limitation noted was the lack of official participant submissions during the EVALITA 2020 campaign, meaning the "state-of-the-art" for this specific task currently rests on the authors' baseline. Future work could involve moving beyond pointer-generator networks to more modern transformer architectures (like GPT-style models) fine-tuned on this dataset. Additionally, the authors suggest that their alignment procedure could be further refined to better disentangle newspaper-specific style from the actual topics being reported.

Summary updated

BibTeX
@inproceedings{mattei2020change,
  author = {Lorenzo De Mattei and Michele Cafagna and Felice Dell'Orletta and Malvina Nissim and Albert Gatt},
  title = {{CHANGE-IT @ EVALITA 2020: Change Headlines, Adapt News, GEnerate (short paper)}},
  booktitle = {EVALITA},
  year = {2020},
  url = {https://openreview.net/forum?id=iEi8TioFnmI},
}
Figure