Discussion

Our study shows that it is possible to build dynamic models for research careers of EPs that render very good adjustments both individually and for groups of researchers. Conversely, we found a lack of a close relationship between investigator experience and his/her production and repercussion, which was not totally unexpected. Final research performance depends on various aspects, including the potential of the research group to which the investigator belongs, membership in multicenter research networks, the specialty or field of research and whether the research is basic or clinical.10 This leads to significant variations between medical disciplines.11 Thus, investigators working in high-activity areas tend to produce more documents, generate more citations and enjoy access to more journals with higher IF ratings than investigators working in more restricted areas of knowledge.12 In this sense, EM is at a disadvantage with respect to other medical specialties, partly because it is a relatively new specialty. For example, in 2011 the JCR only included 24 journals in this specialty, and the highest IF was 4.133, for the Annals of Emergency Medicine, being far below that of the top-ranked journals with IFs that exceed 30 (Science, Nature, Cell, The Lancet) and even 50 (The New England Journal of Medicine). Recently, a direct relationship has been found between the size of the category in which a journal is indexed and the IFs achieved by journals forming this category, placing EM journals (and consequently, EM researchers) at a disadvantage with respect to other specialties.13 In any case, it seems clear that crude authorship production data, without pondering other parameters or contextualising, do not allow accurate assessment of a research career and worth, and in this scenario, dynamic analysis could contribute to a better interpretation of a research career.

In this sense, our dynamic temporary assessment of research activity provides a better picture of research career than static data on production or impact indicators. Analysis of each EP showed good fit in the temporary evolution of production and scientific impact. This growth fit a quadratic model, in many cases almost perfectly, which was particular for each investigator. This applied to all of the indicators analysed, with the best coefficient of determination seen for the h-index. Thus, our hypothesis for future research is that, assuming stable personal and working life conditions of the investigator, it is possible to use this model to predict the future scientific production and quality of a particular author. In fact, if a constant pattern emerges after a certain time, it could be used to predict his or her research output ceiling at the end of a career, or at intermediate intervals. This would help solve the difficult problem of assessing the value of emerging investigators. Specific indicators may well underestimate their true relevance,14 since they are at a distinct disadvantage in terms of career time compared with established investigators. In the opposite sense, the individual model could also serve to detect decreased or lower-than-expected rates of scientific production with respect to those expected by the scores achieved in their former years. All this has implications for resource allocation for research projects or grants, and thus, this hypothesis should be confirmed by further studies.

With respect to making estimates, the present study shows that it is also possible to apply models, although less accurately, to the overall group of EP investigators in our sample. We observed that collective estimation is better achieved by grouping EP researchers according to the intensity on their research (the growth rate, which can be considered as a surrogate of this) rather than their geographic location. Although the subgroups based on the rate of growth were arbitrarily preset according to the rate of increase of the h-index, we believe that, with this approach, the general curves obtained in the present study may serve to evaluate other EP research careers. Indeed, based on our findings, an EP with pronounced research activity but not included in this study would be expected to have an individual growth curve located within the CIs defined for the fast-rate growth group. It is important to recognise that this projection has to be done within the specialty niche because, although the h-index applies to investigators from different specialties, it is clear that each area of knowledge must define the evolution of its own h-index, as shown in previous studies comparing different disciplines.15–17

We recognise that, although the h-index showed the best coefficient of determination in our study, it is a bibliometric index based on quantity (number of citations and papers) and does not directly analyse the quality of the publications. This is a common defect of most of the currently available indexes,18–20 and significant progress in this field is needed over the next few years in order to make it increasingly easier to quantitatively and qualitatively evaluate both, individual and group research activity. On the other hand, our study collected data about the productivity and repercussions of EM researchers based on h-index and IF. These historical tools, however, only measure the impact of a researcher as it pertains to other researchers. They do not measure the impact of a researcher with respect to reader uptake and knowledge translation, which are outcome measures that could, ultimately, be more important for clinical medicine. Tools that do measure uptake—at least to some degree—are now available and are being taken up across the medical world. Related to information exchange through social network services, blogs and podcasts, altmetrics measure the discussion and value of research as it applies to clinicians as well as academicians.21 Academic social networks such as Mendeley, and CiteULike as well as social media are monitored to identify the level of activity related to each article. Future research evaluating the impact and repercussion of research should include these metrics in addition to the h-index and IF.

The present study has limitations. First, there are those inherent to each of the indicators analysed,22 especially because they attach greater importance to senior authors and review articles than to young researchers and single studies. Second, errors in author assignation of papers could have occurred in both directions, mistakenly assigning publications to authors who had not written them (due to homonymous), and dismissing papers because of incorrect spelling of authors. Use of a unique identifier such as the ORCID ID (http://orcid.org/) is recommended by the World Association of Medical Editors to overcome this confusion, but not all authors evaluated in present study had a registered ORCID ID. Thus, despite self-author checking, errors may have crept in but this is unlikely and, in any case, is likely of little consequence. Third, the present study did not investigate self-citation, although analyses to date have not clearly established the importance of this phenomenon.23 ,24 Fourth, the data analysed do not meet criteria of independence due to autocorrelation of the series. Further studies are therefore needed with more complex models, such as Auto Regressive Integrated Moving Average (ARIMA) or others, to validate these hypotheses. Fifth, the relationship between scientific production and author gender has been poorly studied; our work only revealed 18 EP investigators as women, which shows a clear disproportion in favour of men. This is consistent with previous studies conducted worldwide.25 ,26 In the case of the present study, the inclusion of investigators with long careers likely resulted in the low participation of women, as their incorporation into the workforce of medicine in general, and in research in particular, has occurred more recently than for men. Sixth, our strategy for selecting EPs highly involved in research was to pick them from the lists of editorial boards of six of the top-rated EM journals, because we hypothesised that the more prolific and prestigious researchers had to be included among the members of these editorial boards. This was arbitrary and may include a bias since we are aware that many reputed EP researchers are not involved in editorial committees, and those who are members of these boards may actually not be as involved in research as other researchers not evaluated in the present study. On the other hand, our sample was selected to represent relatively high achieving academics, but many reputable clinical academics (who are not represented in our sample) may have alternative trajectories that have not yet been defined. It remains too early, therefore, to use our work to decide that an individual is underperforming. Seventh, to take the first published paper as the beginning of scientific career was also arbitrary and could imply some bias. For example, some EPs could have published as students and then not published again for several years. Although this is not unusual nowadays, we believe that most of the researchers included finished their medical studies more than 20 years ago, at which time research at medical school was not as frequent as at present. Accordingly, we consider this bias unlikely and of very low impact if present. Finally, although we have tried to discuss the results objectively and without any individual personal reference, 6 of the authors of the study were included among the 107 analysed researchers.

In conclusion, the data obtained in the present study show that it is possible to construct individual and group curves of career trajectories for EPs conducting research. These results allow better understanding of the research activity and the scientific impact of this activity in the field of EM worldwide and enables future comparisons with other groups of investigators and future generations of EP researchers.