The textual characteristics of traditional and Open Access scientific journals are similar

Karin Verspoor, K. Bretonnel Cohen, Lawrence HunterBMC Bioinformatics 2009 · DOI · OpenReview

TL;DR

This paper investigates whether Open Access (OA) scientific journals have distinct textual characteristics compared to traditional subscription-based journals, a crucial question for natural language processing (NLP) research that often relies on OA data. The study found no significant structural or semantic differences between the two types of journals, suggesting that current NLP methods developed for OA articles are likely generalizable to the broader biomedical literature.

What Problem Does This Paper Address?

Natural Language Processing (NLP) and text mining in the biomedical domain have increasingly moved from analyzing article abstracts to processing full-text publications. A significant portion of this full-text research utilizes Open Access (OA) articles, largely due to their free and easy availability. This reliance on OA data implicitly assumes that OA articles are representative of the entire biomedical literature. However, potential differences in publication models (e.g., electronic-only, no strict length limits for OA journals) and community perceptions (e.g., higher impact for OA) could lead to textual variations. If OA articles are not representative, NLP models trained on them might perform poorly on traditional journals, leading to wasted resources and flawed scientific conclusions. This paper directly addresses this assumption by comparing the textual characteristics of OA and traditional scientific journals.

Methods

The researchers employed a comparative analysis of linguistic features and semantic content across different corpora.

  • Surface Linguistic Structures: They examined the incidence of phenomena known to affect NLP difficulty, including:
  • Negation (e.g., "no," "not," "n't")
    • Passivization (e.g., "is phosphorylated by")
    • Conjunctions (e.g., "and," "or," "but")
    • Pronominal anaphora (pronoun usage)
    • Sentence length distribution
  • Semantic Content: To assess semantic differences, they used:
  • Kullback-Leibler (KL) Divergence: This measures the difference between probability distributions of words in different corpora. A low KL divergence indicates similar semantic content.
  • Log Likelihood Analysis: This identifies terms that are significantly more or less frequent in one corpus compared to another, helping to pinpoint specific lexical differences.

Datasets

The study utilized four text collections for comparison:

  • CRAFT (Colorado Rich Annotation of Full Text): An Open Access corpus focused on mouse genomics, comprising 97 articles and approximately 750K words, sourced from the PubMed Central Open Access subset.
  • TraJour (Traditional Journals corpus): A corpus of traditional subscription-based journals, topically paralleling CRAFT, consisting of 99 articles and nearly 600K words.
  • Reference (Wall Street Journal corpus): A general news corpus (approx. 1.1 million words) used as a non-biomedical comparison.
  • BioReference: A corpus intended to be representative of general biomedical publications, constructed from a random subsample of the TREC Genomics Corpus (traditional journals) and the PubMed Central Open Access subset. This corpus contained 163 articles and approximately 650K words.

Key Insights and Ideas

  • OA vs. Traditional Journals are Textually Similar: The core finding is that, for the linguistic features examined, OA and traditional biomedical journals are remarkably alike. This is a significant result for the field of BioNLP.
  • Surface Features Don't Differentiate: Common NLP challenges like negation, passives, conjunctions, and pronoun usage occur at similar rates in both OA and traditional journals. Sentence length distributions also show a similar mode.
  • Semantic Overlap is High: KL divergence scores between the OA (CRAFT) and traditional (TraJour) corpora were very low, especially for the most frequent terms, indicating substantial overlap in their vocabularies and semantic content.
  • Differences Lie in Formatting and Specific Entities: When differences did emerge, log likelihood analysis pointed towards formatting artifacts (e.g., "figure," "doi") and specific named entities rather than fundamental differences in scientific discourse.
  • Biomedical Journals Differ from General Text: Both OA and traditional biomedical journals showed clear differences when compared to the Wall Street Journal (Reference corpus), as expected, with higher rates of passives and conjunctions in scientific writing.

Key Results

  • No Significant Linguistic Differences: The incidence of negation, passivization, conjunctions, and pronominal anaphora did not differ significantly between the CRAFT (OA) and TraJour (traditional) corpora.
  • Similar Sentence Lengths: Both CRAFT and TraJour exhibited the same mode for sentence length distribution (0-10 tokens).
  • Low KL Divergence: The KL divergence between CRAFT and TraJour was consistently low, especially for the top 500 most frequent terms, suggesting near-identical probability distributions of words. This divergence was significantly lower than comparisons involving the Reference (WSJ) corpus.
  • Log Likelihood Highlights Formatting: The top terms distinguishing CRAFT from TraJour in log likelihood analysis were largely related to formatting (e.g., "figure," "doi") and specific entities, not broad semantic themes.

Limitations and Future Work

  • Domain Specificity: The study focused on molecular biology and mouse genomics. The authors acknowledge that generalizing these findings to the entire biomedical literature, or all peer-reviewed scientific publications, is an assumption that would require further testing.
  • Qualitative Differences: The paper focused on quantitative linguistic and semantic measures. While they found no significant differences, subtle qualitative differences that might impact NLP applications could still exist.
  • Significance Testing: The authors explicitly state they did not perform formal statistical significance testing, as they were primarily interested in qualitative differences that could impact NLP applications. Minor variations are expected between corpora.
  • Future Work: The authors suggest that further testing across different scientific domains and publication types would be beneficial to confirm the generalizability of their findings.

Summary updated

Abstract

Recent years have seen an increased amount of natural language processing (NLP) work on full text biomedical journal publications. Much of this work is done with Open Access journal articles. Such work assumes that Open Access articles are representative of biomedical publications in general and that methods developed for analysis of Open Access full text publications will generalize to the biomedical literature as a whole. If this assumption is wrong, the cost to the community will be large, including not just wasted resources, but also flawed science. This paper examines that assumption. We collected two sets of documents, one consisting only of Open Access publications and the other consisting only of traditional journal publications. We examined them for differences in surface linguistic structures that have obvious consequences for the ease or difficulty of natural language processing and for differences in semantic content as reflected in lexical items. Regarding surface linguistic structures, we examined the incidence of conjunctions, negation, passives, and pronominal anaphora, and found that the two collections did not differ. We also examined the distribution of sentence lengths and found that both collections were characterized by the same mode. Regarding lexical items, we found that the Kullback-Leibler divergence between the two collections was low, and was lower than the divergence between either collection and a reference corpus. Where small differences did exist, log likelihood analysis showed that they were primarily in the area of formatting and in specific named entities. We did not find structural or semantic differences between the Open Access and traditional journal collections.
BibTeX
@article{verspoor2009textual,
  author = {Karin Verspoor and K. Bretonnel Cohen and Lawrence Hunter},
  title = {{The textual characteristics of traditional and Open Access scientific journals are similar}},
  journal = {BMC Bioinformatics},
  year = {2009},
  doi = {10.1186/1471-2105-10-183},
}

Loading related content...

Figure