Article Text

Download PDFPDF

Analysis of h-index and other bibliometric markers of productivity and repercussion of a selected sample of worldwide emergency medicine researchers
  1. Òscar Miró1,2,
  2. Pablo Burbano2,
  3. Colin A Graham3,
  4. David C Cone4,
  5. James Ducharme5,
  6. Anthony F T Brown6,
  7. Francisco Javier Martín-Sánchez1,7
  1. 1Editor in Chief, Emergencias, Barcelona, Spain
  2. 2Emergency Department, Hospital Clínic, Emergencies: Processes and Pathologies” Research Group, IDIBAPS, Barcelona, Spain
  3. 3Editor in Chief, European Journal of Emergency Medicine, Hong Kong, China
  4. 4Former Editor in Chief, Academic Emergency Medicine, New Haven, Connecticut, USA
  5. 5Editor in Chief, Canadian Journal of Emergency Medicine, Hamilton, Ontario, Canada
  6. 6Former Editor in Chief, Emergency Medicine Australasia, Brisbane, Australia
  7. 7Associate Editor, Emergencias, Madrid, Spain
  1. Correspondence to Dr Òscar Miró, Emergency Department, Hospital Clínic, Villarroel 170, Barcelona 08036, Catalonia, Spain; omiro{at}clinic.cat

Abstract

Objective To explore bibliometric markers in a worldwide sample of emergency physician investigators to define global, continental and individual patterns over time.

Methods We evaluated the number of papers published, citations received, cumulative impact factor and h-index of editorial board members of six international emergency medicine journals. We calculated the individual values for every year of each author's career to evaluate their dynamic evolution. We analysed the results by researcher world area and growth rate.

Results We included 107 researchers (76 American, 21 European and 10 Australasian; 46 slow-rate -group C-, 43 medium-rate -group B- and 18 fast-rate growth -group A-). The median experience was 18 (IQR: 12) years, without subgroups differences. Dynamic analysis over time showed good fit with quadratic function in all individual researchers and for all bibliometric markers (R2: 0.505–0.997), with the h-index achieving the best R2. The combined analysis of the h-index of the 107 investigators also fit the quadratic model (R2=0.49). Analysis by predefined continental and growth-rate subgroups allowed defining specific patterns (R2 between 0.46–0.54 and 0.80–0.86, respectively): by continents, American researchers' h-index increased 0.632 points per year, European 0.417 and Australasian 0.341; by growth rate, researchers from group A, B and C increased 1.239, 0.683 and 0.320, respectively.

Conclusions Dynamic analysis of every individual author indicator over time has a very good fit with a quadratic model, with the h-index achieving the best R2. It is also possible to construct models based on continent and rate of growth that could help to predict future expected outcomes of researchers in a particular subgroup and to classify new emerging researchers by growth rate.

  • research, epidemiology
  • publication
View Full Text

Statistics from Altmetric.com

Key messages

What is already known on this subject?

  • The evolution of research in emergency medicine (EM) has been extensively analysed in recent years, showing that productivity has gradually increased, in parallel with the global growth observed in research activity and scientific production in recent decades. Nonetheless, no work to date has analysed the evolution of bibliometric indicators of emergency physicians (EPs) involved in research, and the evolution of their research careers. A pilot study evaluated the dynamics of research output of 24 Spanish EPs and showed that it is possible to model changes in researcher bibliometric indicators over time that, in turn, could be used in the future to compare the behaviour of these research indicators with those of other research groups.

What this study adds?

  • Data obtained in this work show that it is possible to construct individual and group curves of career trajectories for EPs conducting research. This allows a better understanding of research activity and scientific impact in the field of EM worldwide and enables future comparisons with other groups of investigators and future generations of EP researchers.

Introduction

The evolution of research in emergency medicine (EM) has been extensively analysed in recent years, and has shown a gradual increase in productivity1–5 in parallel with the global growth observed in research activity and scientific production in the last decades.6 Some bibliometric markers have served to better describe the evolution of research and researchers. Among these, two have gained wide popularity. The impact factor (IF), defined by Garfield,7 estimates the quality of journals in which research is published. According to the definition, the IF (which is calculated yearly) for a journal corresponds to the ratio of citations received by that journal during 1 year regarding papers published in the two previous years divided by the number of citeable articles published during these two previous years. On the other hand, the h-index, defined by Hirsch,8 estimates the relevance of researchers and is calculated as the number of articles that have been cited at least that number of times (eg, an h-index of 10 means that the author has 10 papers that have been cited at least 10 times each). Both bibliometric markers are based on the number of papers published and citations and, although they are quantitative in essence, they are usually also used as a proxy of quality, assuming that the greater the number of citations, the better the quality of the paper.

Nonetheless, to date no study has analysed the evolution of the bibliometric indicators of emergency physicians (EPs) involved in research and the evolution of their research careers from an international point of view. A pilot study evaluated the dynamics of research output of 24 Spanish EPs and showed that it is possible to model changes in researcher bibliometric indicators over time which, in turn, can be used in the future to compare the behaviour of these research indicators with those of other research groups.9 In the present study, we explored the behaviour of several bibliometric markers in a worldwide sample of EP investigators to define global, continental and individual patterns over time. We also tried to define the patterns for the best EP researchers in an attempt to predict the foreseeable h-index at the end of their research careers.

Methods

The researchers included in the study were selected from the list of members of the editorial boards of six international journals ranked at Journal Citation Reports (JCR): Annals of Emergency Medicine, Academic Emergency Medicine, Canadian Journal of Emergency Medicine, Emergency Medicine Journal, European Journal of Emergency Medicine and Emergency Medicine Australasia. In January 2013, these lists comprised 126 members. Considering that the journals selected are among the most prestigious in the field of EM, we considered that among these eligible investigators we would find those with a trajectory that was constant, prolific or very relevant in EM. We excluded investigators not working in EM-related fields, as some members of the boards are leading investigators in other areas of medicine, or are statistical or methodological experts providing advice and collaboration to the journal. The tool used to quantify research activity was the Science Citation Index Expanded (SCI-Expanded), which is a database containing 8631 journals, and access to database was performed between 7 and 13 January 2013.

The search was performed by inserting the investigator name in the field for Author (with possible variants, eg, only one or two names and/or forenames). We reviewed documents retrieved by the search engine, and when a homonym was suspected, we repeated the search with the author's address and affiliation details. ORCID ID numbers were not used because not all researchers are registered in this registry. Then, a complete list of the papers retrieved was sent by email to each author, who was specifically asked to authenticate his or her authorship of every work mentioned in the list, in order to eliminate erroneous captures. The final list was then used to obtain each author's bibliometric markers. The items included in the analysis for each author were the papers published up to 31 December 2012. The author's allocation to a specific country or continent was done based on the current workplace, and years of experience in research were counted from the first paper published, which was defined as year 1.

We evaluated four bibliometric markers: one as an indicator of production (number of papers published in the study period) and three as indicators of scientific impact (number of citations received, cumulative IF and h-index). The number of papers published in any category (original, review, editorial, letters), the number of citations received by these papers and the h-index were provided by the SCI-Expanded application. The cumulative IF of the authors was obtained by the sum of the IFs of the journals in which they had published their papers. The IF was taken as the last known value of the journal at the time of the study, published by the JCR for the year 2011 (the last available at the time of the study), regardless of the year in which the paper was published. For these four bibliometric parameters, we computed the final scores obtained for each author at the end of the study (2012). Additionally, for each bibliometric parameter we also calculated the individual values at each year of the author's scientific career. For this purpose, it was necessary to download the database for each author from the SCI-Expanded and manually analyse his or her productivity year by year.

The results are presented as median and IQR and comparisons were performed by the non-parametric Kruskal-Wallis test. To assess the association between quantitative variables, we used second-order polynomial (quadratic) regression models. The degree of association was quantified by R2 (coefficient of determination) and statistical significance by the value of p. Differences with a p value <0.05 were considered to be statistically significant. We decided a priori to analyse results by world areas, dividing geographic localisations into three zones: Europe, America and Australasia (for simplification purposes, we included Hong Kong and Singapore in this group). We also analysed the bibliometric behaviour of the researchers according to the growth rates of the authors' bibliometric parameters. For this purpose, we used the slope of the regression line of the linear model for the h-index and classified authors into three groups: group C, B and A having growth rates of <0.5, between 0.5 and 1 and >1 point of h-index per year. Data analysis was performed using the SPSS V.18. The study was approved by the Clinical Research Ethics Committee of the Hospital Clínic, Barcelona. Since the data analysed are public and can be collected from the aforementioned database, we were waivered to obtain investigators’ permission to be included in the study, guaranteeing that particular indicators would be presented anonymously.

Results

Of the 126 eligible researchers, 19 were excluded (6 not working in EDs, 13 not returning the check list of published papers). Thus, a total of 107 researchers were included: 76 North American researchers (62 USA, 14 Canada), 21 European researchers (13 UK, 3 France, 2 Spain, 1 Holland, 1 Sweden, 1 Belgium) and 10 Australasian researchers (5 Australia, 3 Hong Kong, China, 1 Singapore, 1 New Zealand) (box 1). The median experience in research was 18 (IQR 12) years, ranging from 3 to 39, and 2116 individual year periods were analysed. There was a weak (R2: 0.10–0.22), although statistically significant, association between the years involved in research and all the bibliometric parameters, with the greatest determination coefficient being seen for the h-index (figure 1).

Box 1

Researchers included in the analysis of present study, classified by continents (bold letters denote female researchers)

America

▸ Callaham M (USA), Diercks DB (USA), Houry DE (USA), Hollander JE (USA), Rinnert KJ, Schriger DL (USA), Talan DA (USA), Yealy DM (USA), Burstein JL (USA), Wolfson AB (USA), Brice JH (USA), Moran GJ (USA), Cydulka RK (USA), Hendey GW (USA), Todd KH (USA), Knopp RK (USA), Levitan RM (USA), Marill KA (USA), Delbridge TR (USA), Sarko J (USA), Sztajnkrycer MD (USA), Mower WR (USA), Milling TJ (USA), Moorhead JC (USA), Zalenski RJ (USA), De Maio VJ (USA), Atzema C (CA), Schull MJ (CA), Gorelick MH (USA), Jagoda AS (USA), Cone DC (USA), HaUSAwald M (USA), Baren JM (USA), Newgard CD (USA), Conners GP (USA), Gaddis GM (USA), Costantino TG (USA), Alpern E (USA), Courtney DM (USA), Baumann BM (USA), Gratton MC (USA), Bonsu B (USA), Griffey RT (USA), Callaway CW (USA), Hiestand BC (USA), Choo EK (USA), Meisel ZF (USA), O'Neil BJ (USA), Mello MJ (USA), Stahmer SA (USA), Merchant RC (USA), Theodoro DL (USA), Reardon RF (USA), Ufberg JW (USA), Sinert R (USA), Wilber ST (USA), Mycyk MB (USA), Yarris LM (USA), Zehtabchi S (USA), Joing S (USA), Sokolove PE (USA), Delorio NM (USA), Fernandes CMB (CA), Ducharme J (CA), Patocka C (CA), Abu-Laban R (CA), Morrison LJ (CA), Sivilotti M (CA), Afilalo M (CA), Stiell IG (CA), Sherbino J (CA), Kissoon N (CA), Zed PJ (CA), Travers AH (CA), Tintinalli J (USA), Dieckmann RA (USA)

Europe

▸ Mackway-Jones K (UK), Goodacre S (UK), Maconochie I (UK), Heyworth J (UK), Baombe JP (UK), Alao D (UK), Wardrope J (UK), Challen K (UK), Ameh V (UK), Coats T (UK), Bleetman A (UK), Foex B (UK), Maskery N (UK), Bierens J (HO), Bounes V (FR), Duchateau FX (FR), Castren M (SW), Miró O (SP), Hubloue I (BE), Poloujadoff MP (FR), Martín-Sánchez FJ (SP)

Australasia (plus Hong Kong and Singapore)

▸ Ardagh M (NZ), Graham CA (CH), Brown AFT (AU), Holdgate A (AU), Jelinek GA (AU), Cocks RA (CH), Dziukas L (AU), Rainer TH (CH), Fulde G (AU), Seow E (SI)

AU, Australia; BE, Belgium; CA, Canada; CH, China; HO, Holland; FR, France; NZ, New Zealand; SI, Singapore; SP, Spain; SW, Sweden.

Figure 1

Relationship between emergency physician researchers' experience (years involved in research) and outcomes in production (number of published papers) and the repercussion (number of citations, cumulated impact factor and h-index) of their research.

No statistical differences were found in experience or bibliometric outcomes according to the researchers' continents (table 1). When researchers were divided according to their growth rates, 46 (43%) were allocated to the group C (28 American, 13 European, 5 Australasian), 43 (40%) to group B (35 American, 4 European, 4 Australasian) and 18 (17%) to group A (13 American, 4 European, 1 Australasian). Despite no differences in experience among the three groups, there were significant differences in all the parameters analysed on following the scientific careers of the researchers from the groups C, B and A in an incremental order of production and repercussion (table 1).

Table 1

Analysis of the experience and the four bibliometric outcomes, as a whole and according to the researchers' continental origin and their rate of growth

Dynamic analysis of progression in productivity and repercussion over time for each individual researcher showed a good fit with the quadratic model for all individual researchers and for all bibliometric parameters. For example, figure 2 shows these fits for just one of the investigators included in the present study. Among the bibliometric parameters, the best R2 values for the 107 investigators were obtained for the h-index (median of R2: 0.976 (IQR: 0.027)).

Figure 2

An example of the dynamic evolution of the four bibliometric parameters evaluated in an emergency physician career.

Since the h-index was the bibliometric parameter with the greatest coefficient of determination (R2) with respect to researcher career evolution, we used it for the analysis of pooled data of all the investigators together. The combined analysis of the 107 investigators showed a statistically significant fit (R2=0.49), although this adjustment was not as good as that seen for individual researchers (figure 3). When analysed by continents, we found a very similar coefficient of determination for all of the researchers (R2: 0.46–0.54), which was also very similar to that observed for the whole cohort of researchers. The greatest rate of h-index increment was observed in the American researchers (0.632 points per year with the linear model adjustment), followed by the European researchers (0.417) and the Australasian researchers (0.341) (figure 3). On dividing the researchers according to their growth rates, we found that the coefficients of determination were very similar among the three subgroups, but were greater for every subgroup than those observed in the analysis by continents (R2: 0.79–0.86). Annual increments of the h-index were 0.320, 0.683 and 1.239 for researchers allocated to the groups C, B and A, respectively (figure 3). With the data of the researchers of group A, those with the fastest rate of growth, we built the quadratic model to predict the h-index for the high-performance EP researchers along a 50-year course of research, which we considered the usual time that a reputed research career lasts in regular conditions (figure 4).

Figure 3

Dotted plot and regression lines (with 95% CI for individuals) for quadratic models of the researchers' careers obtained by analysing the relationship between the experience of the researcher (years involved in research) and the h-index. The upper line corresponds to analysis of all the researchers together, the second line depicts the dynamic analysis of subgroups according to the continental affiliation of the researcher (for simplicity, Australasia comprises Hong Kong and Singapore) and the third line depicts the dynamic analysis according to researcher rate of growth.

Figure 4

Expected h-index over time for emergency physician researchers showing the fastest rate of growth (group A) in their research careers, according to the quadratic model (with 99% CI) derived from the present study.

Discussion

Our study shows that it is possible to build dynamic models for research careers of EPs that render very good adjustments both individually and for groups of researchers. Conversely, we found a lack of a close relationship between investigator experience and his/her production and repercussion, which was not totally unexpected. Final research performance depends on various aspects, including the potential of the research group to which the investigator belongs, membership in multicenter research networks, the specialty or field of research and whether the research is basic or clinical.10 This leads to significant variations between medical disciplines.11 Thus, investigators working in high-activity areas tend to produce more documents, generate more citations and enjoy access to more journals with higher IF ratings than investigators working in more restricted areas of knowledge.12 In this sense, EM is at a disadvantage with respect to other medical specialties, partly because it is a relatively new specialty. For example, in 2011 the JCR only included 24 journals in this specialty, and the highest IF was 4.133, for the Annals of Emergency Medicine, being far below that of the top-ranked journals with IFs that exceed 30 (Science, Nature, Cell, The Lancet) and even 50 (The New England Journal of Medicine). Recently, a direct relationship has been found between the size of the category in which a journal is indexed and the IFs achieved by journals forming this category, placing EM journals (and consequently, EM researchers) at a disadvantage with respect to other specialties.13 In any case, it seems clear that crude authorship production data, without pondering other parameters or contextualising, do not allow accurate assessment of a research career and worth, and in this scenario, dynamic analysis could contribute to a better interpretation of a research career.

In this sense, our dynamic temporary assessment of research activity provides a better picture of research career than static data on production or impact indicators. Analysis of each EP showed good fit in the temporary evolution of production and scientific impact. This growth fit a quadratic model, in many cases almost perfectly, which was particular for each investigator. This applied to all of the indicators analysed, with the best coefficient of determination seen for the h-index. Thus, our hypothesis for future research is that, assuming stable personal and working life conditions of the investigator, it is possible to use this model to predict the future scientific production and quality of a particular author. In fact, if a constant pattern emerges after a certain time, it could be used to predict his or her research output ceiling at the end of a career, or at intermediate intervals. This would help solve the difficult problem of assessing the value of emerging investigators. Specific indicators may well underestimate their true relevance,14 since they are at a distinct disadvantage in terms of career time compared with established investigators. In the opposite sense, the individual model could also serve to detect decreased or lower-than-expected rates of scientific production with respect to those expected by the scores achieved in their former years. All this has implications for resource allocation for research projects or grants, and thus, this hypothesis should be confirmed by further studies.

With respect to making estimates, the present study shows that it is also possible to apply models, although less accurately, to the overall group of EP investigators in our sample. We observed that collective estimation is better achieved by grouping EP researchers according to the intensity on their research (the growth rate, which can be considered as a surrogate of this) rather than their geographic location. Although the subgroups based on the rate of growth were arbitrarily preset according to the rate of increase of the h-index, we believe that, with this approach, the general curves obtained in the present study may serve to evaluate other EP research careers. Indeed, based on our findings, an EP with pronounced research activity but not included in this study would be expected to have an individual growth curve located within the CIs defined for the fast-rate growth group. It is important to recognise that this projection has to be done within the specialty niche because, although the h-index applies to investigators from different specialties, it is clear that each area of knowledge must define the evolution of its own h-index, as shown in previous studies comparing different disciplines.15–17

We recognise that, although the h-index showed the best coefficient of determination in our study, it is a bibliometric index based on quantity (number of citations and papers) and does not directly analyse the quality of the publications. This is a common defect of most of the currently available indexes,18–20 and significant progress in this field is needed over the next few years in order to make it increasingly easier to quantitatively and qualitatively evaluate both, individual and group research activity. On the other hand, our study collected data about the productivity and repercussions of EM researchers based on h-index and IF. These historical tools, however, only measure the impact of a researcher as it pertains to other researchers. They do not measure the impact of a researcher with respect to reader uptake and knowledge translation, which are outcome measures that could, ultimately, be more important for clinical medicine. Tools that do measure uptake—at least to some degree—are now available and are being taken up across the medical world. Related to information exchange through social network services, blogs and podcasts, altmetrics measure the discussion and value of research as it applies to clinicians as well as academicians.21 Academic social networks such as Mendeley, and CiteULike as well as social media are monitored to identify the level of activity related to each article. Future research evaluating the impact and repercussion of research should include these metrics in addition to the h-index and IF.

The present study has limitations. First, there are those inherent to each of the indicators analysed,22 especially because they attach greater importance to senior authors and review articles than to young researchers and single studies. Second, errors in author assignation of papers could have occurred in both directions, mistakenly assigning publications to authors who had not written them (due to homonymous), and dismissing papers because of incorrect spelling of authors. Use of a unique identifier such as the ORCID ID (http://orcid.org/) is recommended by the World Association of Medical Editors to overcome this confusion, but not all authors evaluated in present study had a registered ORCID ID. Thus, despite self-author checking, errors may have crept in but this is unlikely and, in any case, is likely of little consequence. Third, the present study did not investigate self-citation, although analyses to date have not clearly established the importance of this phenomenon.23 ,24 Fourth, the data analysed do not meet criteria of independence due to autocorrelation of the series. Further studies are therefore needed with more complex models, such as Auto Regressive Integrated Moving Average (ARIMA) or others, to validate these hypotheses. Fifth, the relationship between scientific production and author gender has been poorly studied; our work only revealed 18 EP investigators as women, which shows a clear disproportion in favour of men. This is consistent with previous studies conducted worldwide.25 ,26 In the case of the present study, the inclusion of investigators with long careers likely resulted in the low participation of women, as their incorporation into the workforce of medicine in general, and in research in particular, has occurred more recently than for men. Sixth, our strategy for selecting EPs highly involved in research was to pick them from the lists of editorial boards of six of the top-rated EM journals, because we hypothesised that the more prolific and prestigious researchers had to be included among the members of these editorial boards. This was arbitrary and may include a bias since we are aware that many reputed EP researchers are not involved in editorial committees, and those who are members of these boards may actually not be as involved in research as other researchers not evaluated in the present study. On the other hand, our sample was selected to represent relatively high achieving academics, but many reputable clinical academics (who are not represented in our sample) may have alternative trajectories that have not yet been defined. It remains too early, therefore, to use our work to decide that an individual is underperforming. Seventh, to take the first published paper as the beginning of scientific career was also arbitrary and could imply some bias. For example, some EPs could have published as students and then not published again for several years. Although this is not unusual nowadays, we believe that most of the researchers included finished their medical studies more than 20 years ago, at which time research at medical school was not as frequent as at present. Accordingly, we consider this bias unlikely and of very low impact if present. Finally, although we have tried to discuss the results objectively and without any individual personal reference, 6 of the authors of the study were included among the 107 analysed researchers.

In conclusion, the data obtained in the present study show that it is possible to construct individual and group curves of career trajectories for EPs conducting research. These results allow better understanding of the research activity and the scientific impact of this activity in the field of EM worldwide and enables future comparisons with other groups of investigators and future generations of EP researchers.

Acknowledgments

The authors thank Steve Goodacre, Ellen Weber and David Schriger for their valuable and insightful comments in a previous version of the final manuscript.

References

View Abstract

Footnotes

  • Contributors OM, PB, FJM-S designed the study. CAG, DCC, JD, AFTB discussed the design and add valuable comments. OM and PB did the recruitment of data. PB built up the database. OM, PB, FJM-S made statistical analysis. CAG, DCC, JD, AFTB discussed statistical results. OM wrote first manuscript draft. All authors read and made comments on such a draft. Figures and tables were prepared by OM and PB.

  • Competing interests None declared.

  • Ethics approval Clinical Research Ethics Committee of Hospital Clínic, Barcelona, Spain.

  • Provenance and peer review Not commissioned; externally peer reviewed.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.