Are citation counts of articles related to outcomes and legacies of authors and journal editors? (CAROL): a cohort study

Our study aimed to investigate the association between quantitative characteristics of editors or corresponding authors and the number of citations each article received. To avoid bias when analyzing journals with varying impact factors, we selected one journal (PLOS Medicine) where information about the editor and corresponding author is publicly available and then analyzed the published articles. To increase the reliability of the study, we also selected only articles published in 2018 and 2019, before the beginning of the COVID-19 pandemic, to exclude the influence of abnormal citation patterns due to the pandemic.[13] Next, our study included several factors acquired from Web of Science as follows: citations of publications, H-index, number of publications, duration of research career, and number of publications from the affiliation of each corresponding author and their handling editors. Additionally, we investigate the gross national income per capita of the affiliation country for each corresponding author and editor. These variables were chosen to comprehensively reflect the various factors that can affect the number of citations of an article. This study systematically screened articles (n=428) published in the specified years by searching the Web of Science. Allowing for duplication in editors and corresponding authors, 396 independent articles were ultimately selected for this study, all of which were publicly accessible and included a range of variables that could affect the number of citations an article received.

To reflect the importance of the Christmas date, our study categorized the top 12.24% of all articles (n=49) with the most citations as “super-citations” and the rest as “normal-citations” (n=347). This binary classification is important to distinguish the influence of articles based on the number of citations and to analyze the characteristics of highly cited articles. This categorization was utilized to compare which of the many variables of editors and corresponding authors have a higher association with the super-citations by examining the difference between highly and normally cited articles.

Our study initially performed the Shapiro-Wilk test and D’Agostino’s K² test to check the normality of each variable.[14] The Shapiro-Wilk test is particularly sensitive to small and medium sample sizes and checks to see if variables follow a normal distribution, while D’Agostino’s K² test evaluates skewness and kurtosis to see if the data set comes from a normal distribution.[14] Since these tests verified that not all variables met normality, we used Spearman’s rank correlation to examine the correlation between the continuous variables. Unlike Pearson’s correlation, which assumes a normal distribution of variables, Spearman’s rank correlation is a non-parametric measure that evaluates how well the relationship between two variables can be described using a monotonic function that fits non-normally distributed data.[15] Finally, we performed a Mann-Whitney U-test to assess the statistical difference between the two independent samples: the super-citation and normal-citation groups.[16] The Mann-Whitney U-test is a non-parametric test used when data does not follow a normal distribution. It focuses on the median between two groups and evaluates the difference between the two groups based on the ranking of the data points.[16] The SAS software (version 9.4; SAS Institute, Cary, NC, USA) was used for statistical analyses with a two-sided test. P ≤ 0.05 was considered statistically significant.

This Citation counts of Articles Related to Outcomes and Legacies of authors and journal editors (CAROL) study examined the statistical association between the number of citations of published articles and various factors of corresponding authors and editors. We found that the quantitative characteristics of the corresponding author or editor were weakly correlated with the total number of citations as a continuous variable. However, a notable finding was that the H-index of the editor had a significant association with super-citations when the top 12.24% of super-citations were binary and categorized separately from the other normal citations. This suggests that articles managed by handling editors who maintain consistently high research quality over a long period of time are more likely to be breakthrough and widely popular.

Various studies have been conducted to predict the future citations of articles.[2-12] Some studies utilizing machine learning have successfully predicted future citations by analyzing the text of manuscripts.[5, 6] However, most of the previous studies have been limited to non-medical fields or specific subfields within the medical field. Additionally, these studies primarily focused on the characteristics of the articles, such as the topic, research design, and number of pages, rather than the human characteristics of the editors and authors. Given the recent emphasis on patient-centered medical care, we aimed to focus more on the characteristics and capabilities of editors and authors to conduct more human-centered research.[17]

This study was inspired by a landmark cohort study of BMJ editors (n=10) published on Christmas Day 2022. The prior study showed that even editors of leading medical journals like the BMJ struggle to predict which articles will be highly cited. Nonetheless, our study exhibited that editors with rich research experience are significantly more likely to handle articles that achieve super-citations. At first glance, these two findings may seem contradictory, but this is likely due to differences in the underlying study design. Previous cohort studies have directly asked editors to predict the future citations of specific articles. Our study differs in that citation potential was not considered by editors at the time of editing, and future citations were examined retrospectively. Our design better reflects the real-world situation because, in the actual manuscript review process, editors do not consciously select articles solely based on citation potential but rather comprehensively evaluate the scientific importance of a manuscript based on their accumulated research experience.[1, 18]

The strength of our study lies in its analysis of a large-scale scholar cohort, rigorously examining the influence of corresponding authors’ and editors’ quantitative research capabilities and their affiliations on research impact, based on robust statistical methodologies.

Our study has several limitations. First, most journals did not disclose the editor of each article, limiting the number of journals for which we could include in our study. However, this limitation had the advantage of controlling for confounding factors, allowing us to make the impact factors of the articles in our study even. Second, even if articles are published in the same year, the number of citations will inevitably vary between articles published earlier in the year and those published later. Third, if the editor or corresponding author has incorrect author name information or is published under multiple names, the citation count may not be accurate. To address this, we used an OR condition on possible name combinations to ensure that as many records of the editor or corresponding author were included as possible.

Some scientific journals prefer to achieve a high impact factor, as it has been considered a leading, albeit controversial, indicator of a journal’s relative importance in recent decades.[19-23] The H-index can be achieved only by consistently publishing citable articles, being less influenced by a small number of over- or under-cited outliers, and allowing enough time for citations to accumulate.[24] Our results suggest that if a journal aims to increase its impact factor, it can do so by hiring scholars with a high H-index and a strong long-term research experience as editors.[24] However, as the editorial decision-making process is complex and involves diverse factors,[25] it is important to evaluate not only the statistical value of the editor but also whether the editor possesses the appropriate competencies for the role.[26-28]

Our results also show that a corresponding author’s research experience, number of publications, and citations of publications are only weakly positively correlated with future citations. This indicates that these factors do not guarantee super citations or higher research impact. Consequently, this suggests that editors should not assume that corresponding authors with high reputations will necessarily produce a high impact article when reviewing new manuscripts. Finally, in line with previous studies, high citations are not something that editors can intentionally secure but rather are the result of scholars with high research capabilities focusing on the scientific rigor and completeness of manuscripts based on their experience.[1, 26]