Article Text

Reliability and validity of a new French-language triage algorithm: the ELISA scale
  1. Jerôme Jobé1,
  2. Alexandre Ghuysen1,
  3. Paul Gérard2,
  4. Gary Hartstein1,
  5. Vincent D'Orio1
  1. 1Emergency Department, University Hospital of Liege, Liège, Belgium
  2. 2Statistics Department, University of Liège, Liège, Belgium
  1. Correspondence to Dr Jérôme Jobé, Emergency Department, University Hospital of Liège, CHU Sart Tilman (B35), Rue de l'hôpital, Liège 4000, Belgium; Jerome.jobe{at}


Objective Overcrowding in emergency departments (ED) leads to reductions in quality of care. Consequently, several different triage tools have been developed to prioritise patient intake. Differences in emergency medical services in different countries have limited the generalisation of pre-existing triage systems; for this reason, specific algorithms corresponding to local characteristics are needed. Accordingly, we developed a specific French-language triage system named Echelle Liégeoise d'Index de Sévérité à l'Admission (ELISA). This study tested its validity and efficiency.

Methods ELISA is a five-category nursing triage algorithm. Intrarater agreement was tested by comparing triage levels attributed to the same clinical scenarios at two different times. Interrater agreement was investigated by comparing triage categories attributed to clinical cases by different triage nurses. Finally, validity was estimated by studying the correlations between the triage ranking assigned by the nurse and actual resource consumption and patient outcome.

Results The distribution of the difference between nurse classification at the two times was statistically unrelated to which nurse carried out the evaluation. Regarding interrater agreement, assigned classifications were compared to the reference assignment. Cohen's κ coefficient revealed an almost perfect agreement between classification by nurses and the reference. Finally, statistical analysis revealed a strong relation between ELISA and the overall need for supplementary clinical testing. Outcomes were also significantly correlated with ELISA.

Conclusions The need for a specific, French-language triage tool in our ED led us to develop a new triage scale. This study demonstrates that the scale is a valid triage tool with high interrater and intrarater agreement and considerable efficiency.

  • triage

Statistics from

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.


For some decades, emergency department (ED) overcrowding has been recognised as a major international concern, leading to obvious operational and logistic problems with possible detrimental ethical and moral consequences. Physicians working in a crowed ED may perform less detailed clinical evaluations or make hurried treatment decisions. Under such conditions, emergency medicine may be a speciality particularly prone to errors.1 Indeed, two recent studies have demonstrated an association between ED crowding and increased inpatient mortality.2 ,3

In the beginning of the 19th century, triage was developed on the Napoleonic battlefields, with the aim of providing immediate care to the most critically injured soldiers.1 In a modern extension of this concept, triage systems have been adapted to ED conditions with the aim of getting the right patient to the right resources, at the right place and at the right time. Accordingly, several English-language triage algorithms have been developed: the Manchester triage scale4 in the UK, the Australasian triage scale5 in Australia, the Canadian triage acuity scale6 in Canada, the emergency severity index (ESI)7 in the USA and the South Africa triage scale8 in South Africa. From the beginning, such five-category triage systems were shown to be safe and to provide significant discrimination and reliability, as well as high sensitivity and specificity.9

However, wider implementation of these triage scales throughout Europe has been limited by language constraints. Indeed, direct translation may introduce several errors because English-language triage scales, but not their translations, have been validated in English only. Adaptation has also been made difficult because of major differences in prehospital medical care. In Europe, prehospital care is usually provided by physicians, as opposed to paramedic-based systems used elsewhere. Under these circumstances, nursing triage in Europe may fail when using scales developed in countries where prehospital management is based on a ‘scoop and run’, rather than a ‘stay and play’, strategy.1 ,10 ,11

French-language triage systems are scarce, with the exception of the French emergency nurses classification at hospital (FRENCH) triage tool, a reliable and precise triage system developed at St Louis Hospital in Paris.11 However, in the FRENCH algorithm, nurses have to select among a discouraging number of determinants also including a suspected diagnosis. This situation is further complicated by the definition of triage modulators. This level of complexity, and the need for onerous paramedical team training may have limited the implementation of this scale.

In most ED, the walk-in population tends to be separated between stable patients, able to walk and wait in waiting rooms, and more severely ill patients whose conditions immediately require being supine in a bed. Such local specificities, language constraints and the need for an easy-to-use French-language triage algorithm led us to develop a new triage instrument. We therefore devised a five-level acuity measure called Echelle Liégeoise d'Index de Sévérité à l'Admission (ELISA) to meet the needs of nurses. This scale aimed at including not only gathering data in a short period of time but also combining them into a coherent clinical judgement that purposely omits any diagnostic hypotheses.


The purpose of this study was to investigate the reliability of this new scale and to validate its efficacy in predicting patient resource consumption, the need for admission and outcome. In particular, we planned to measure interrater and intrarater agreement regarding triage assessment and the probability of admission, the urgency to see a physician, the need for monitoring and for rapid diagnostic tests.12–18

Material and methods

The ELISA scale

Initially, the triage nurse's assessment in our department was based on the nurse's intuitive decision as to whether a patient required emergent care and/or of the potential for rapid deterioration in order to categorise a given patient as ‘urgent’ or ‘non-urgent’. The ELISA scale developed as an algorithm based on simple flow charts (including vital signs and selected patient historical data) to categorise walk-in patients into categories ranging from emergent (ELISA1) to non-urgent (ELISA5) need for care. Each category was associated with a pre-established maximum time delay to first medical contact (table 1).

Table 1

ELISA category and corresponding maximum acceptable time to first medical contact

Using these guidelines, we developed our specific algorithm for classifying most common chief complaints and symptoms of adults who are admitted to the ED.

ELISA was developed using a Delphi method19 by a panel of emergency physicians from our institution.

The first step in the assessment is the identification of patients able to walk and speak, who can be directed to ambulatory areas, and patients whose condition requires them to be supine. Then, the triage nurse distinguishes trauma from non-trauma patients and uses simple flow charts, based on the main complaint and vital signs, to assign a triage category. No reference is made to potential diagnosis or further resource consumption (resources are listed in tables 24 below).

Table 2

Relationship between ELISA levels and resource consumption

Table 3

Distribution of therapeutic procedures according to ELISA score

Table 4

Relation between ELISA categories and outcomes

The triage results are expressed in terms of emergency level, as well as the preferred location for further care (ambulatory area, waiting room, immediate placement on a gurney and intake to the non-ambulatory area of the ED, resuscitation unit). The complete ELISA algorithm is available at and in supplementary appendix 1 (available online only).

ELISA is very easy to teach. Full training for our nursing staff requires only 2 h of theoretical teaching and 2 days of practice under the supervision of a trained nurse. Mastery of the algorithm was assessed by ongoing evaluation. Indeed, after triage on admission, patients are oriented to different sectors of the ED where they are cared for by other nurses, offering the opportunity for continuous coaching of less experienced triage nurses and triage corrections when appropriate.

When making the assessment, the criteria are to be used as guidelines rather than rigid rules. As a result, the triage nurse can upgrade the patient's triage level to a higher intensity according to her subjective impression. However, downgrading to lesser intensity than is assigned by the algorithm is not allowed. In case of doubt, the triage nurse can also ask for medical advice by phone or request a physician in the triage room.

Study setting and population

The study site was a 700-bed universitary hospital. The ED admits more than 42 000 patients each year with an ED hospital admission rate of 30%. Obstetric patients with near-term pregnancy and paediatric patients with unstable conditions were excluded, as these patients are usually brought to another site of our hospital complex, based on dispatching algorithms of the emergency medical services call centre.

The triage nurse is based in a room next to the main entrance of the ED. The triage nurse is responsible for controlling patient flow through the various areas of the department.

All nurses participating in the study were specialised emergency care nurses with previous triage experience. Participation was on a voluntary basis after comprehensive explanation of the aims of the study.

Sixteen nurses participated in the study (women, n=8 and men, n=8). The mean age of nurses was 31 years20 and the mean professional experience was 8 years.

We studied a population-based convenience sample of patients triaged at the study site daily over a 3-month period.

The design of the study was approved by the hospital's human subject ethical review committee.

Study protocol

This was a two-phase experimental study. Phase one of the study was focused on the assessment of the reliability of the algorithm, using analysis of written scenarios derived from real clinical cases chosen by a group of senior ED physicians. The aim was to present a representative mix of cases with an equal number of patients in each triage level. The second phase consisted of a retrospective categorisation of a group of patients whose discharge diagnosis was known by study nurses.

Intrarater agreement was tested by comparing the triage levels attributed to the same clinical scenarios at two different times by the same nurse (n=10): early in the morning (T1) and late in the evening (T2) another day. Each nurse triaged the same 30 patients (six from each triage level) and each patient was sorted at the two time points. Interrater agreement was investigated by comparing triage levels assigned by nurses (n=9) to 100 clinical cases (20 from each level triage), also presented as written scenarios. Cases were presented randomly and the nurses were blinded to each other's triage assessment.

Finally, validity was estimated by studying the correlations between the assigned triage ranking and resource consumption, as well as ED stay and outcome in a retrospective investigation of 544 consecutive admissions over a 1-week period.


The following variables were extracted from the records: demographic information, triage ranking, resources used and therapeutic procedures. Resources were recorded according to previous publications on triage scales.7

A composite resource intensity variable was also defined and scored as follows: 0=no resources used, 1=any one resource used, and many=two or more resources used.

Therapeutic procedures were classified as basic procedures, specialised procedures or resuscitation manoeuvres.

Finally, the outcome at 30 days after ED admission was recorded.

Statistical analysis

Intrarater agreement

We compared the classification at T1 and T2 with the null hypothesis of no difference and determined if differences were nurse-dependent according to the Kruskal–Wallis test.

Interrater agreement

The assigned classifications were compared to the reference assignment and analysed with Cohen's κ coefficient.


Each resource was considered as a binary variable and was analysed using four statistical tests: χ2, Wald Wolfowitz, Kolmogorov–Smirnov and Mann–Whitney.

To study the relationship between triage and severity, we calculated the area under the receiver operating characteristic curve for the prediction of hospitalisation, intensive care unit (ICU) admission and death.



The mean age of the study population was 45 years. According to triage categorisation, 6.25% (n=34) patients were categorised as ELISA1, 12.5% (n=68) ELISA2, 41% (n=223) ELISA3, 36.6% (n=199) ELISA4 and 3.65% (n=20) ELISA5. Regarding outcome, 164 patients (33.6%) were hospitalised and 19 (3.5%) were admitted to the ICU. Twelve (2.2%) patients eventually died. Causes of death were cardiac (n=3), pulmonary (n=3), haemorrhagic shock (n=2), stroke (n=2), multi-organ failure (n=1) and terminal oncological disease (n=1).


Intrarater agreement

Neither the distribution of the difference nor the median difference between nurse classification at T1 and classification at T2 significantly depended on the time the nurse carried out the evaluation (Kruskal–Wallis test p=0.08 and median test p=0.3487). This result allowed pooling of the data and testing to determine if the mean of the difference was different from zero. Accordingly, the Sign test confirmed that classification at T1 was not different from classification at T2 (p=0.581).

Interrater agreement

A variance component model involving two random factors (nurse and patient) was used to analyse the classification data. Using this model, the interrater correlation coefficient between two nurses was calculated and was found to be equal to 0.995. The component of variance due to the nurse factor appears to represent 0.05% of the total variance and 1% of the residual variance.

In addition, the linearly weighted κ coefficient between each nurse and the reference varied from 0.89 to 0.99 for each of the nurses.

Assigned classifications were compared to the reference assignment. Cohen's κ coefficient revealed an almost perfect agreement between classification by the group of nurses and the reference, with a linearly weighted Cohen's κ coefficient of 0.99.



Table 2 illustrates the relationship between ELISA levels and resource consumption. Resource consumption was statistically significantly proportional to the severity of the patients as reflected by the triage category if specialist consultations, serologies, basic x-rays and endoscopies are excluded from the resource list.

Figure 1 shows the number of resources consumed according to the ELISA level.

Figure 1

Resources consumed according to the Echelle Liégeoise d'Index de Sévérité à l'Admission (ELISA) level.

Therapeutic procedures

Table 3 shows the distribution of therapeutic procedures according to the ELISA score. This distribution of therapeutic procedures was statistically significantly proportional to the severity of the patients as reflected by the triage category except for tranfusions (blood, platelet, plasma), wound disclosure and surgery. Because too few of the following procedures occurred in our sample, no conclusion could be reached: splints, arm slings, reductions of luxations, drainage of ascites, pleural effusions and lumbar puncture, chest tube insertion, cardiopulmonary resuscitation, mechanical ventilation and defibrillation because of an insufficient number of events.


As illustrated in table 4, hospitalisation, ICU admissions and death were significantly related to a greater severity of the ELISA level, while discharge was more frequent in patients with higher, less severe, ELISA levels.

Areas under the receiver operating characteristic curve for the prediction of discharge, hospital admission, ICU admission and death, as a function of triage, were 0.77 for discharge, 0.77 for hospitalisation, 0.94 for ICU admission and 0.74 for death. For all, p values were <0.05 (figure 2).

Figure 2

Areas under the receiver operating characteristic (ROC) curve for the prediction of discharge, hospital admission, intensive care unit (ICU) admission and death.


In this study, we aimed to evaluate the effectiveness of our triage algorithm. The benchmark should ideally be the relation of triage category to the outcome of the illness episode.22 However, there is currently no gold standard of triage accuracy.23 ,24 As a consequence, most criteria used in the literature to assess triage protocols include simplicity, reliability and validity. Reliability describes the ability of the algorithm to be replicated by the same nurse under different circumstances or to be replicated under similar circumstances by different nurses. We have shown that the ELISA algorithm offers good reproducibility, as confirmed by the high interrater agreement. Furthermore, most studies examining the interrater reliability of triage protocols (using either real cases or clinical scenarios to test the consistency of triage assessment) have demonstrated weighted κ scores ranging from 0.69 to 0.87.11 From that perspective, the weighted κ score of 0.995 obtained with the ELISA scale confirms considerable reproducibility. The high level of intrarater agreement may also be a result of the consistent level of training for nurses. On the other hand, it may also be the result of the triage system itself, which includes the nurse's ability to change the severity score according to their own visual assessment. The ability of the nurses to upgrade on a subjective basis did not underpin any of the outcomes at ELISA1 or ELISA2. From a theoretical point of view, assessment of validity includes both construct and outcome validity. Construct validity addresses the logic of the structure and its application, and involves comparison with other measures of relevance such as indices of urgency or severity. Construct validity is not directly observable. Outcome validity describes the ability of the measure to predict outcomes accurately. The ability of a triage scale to predict resource consumption and hospital admission has been considered to be a convenient marker of validity.12 As a consequence, the ESI triage algorithm, which classifies those patients who can wait into groups based on the predicted need for resources, has been proposed as a potential candidate for a standard triage acuity measure for the ED.20

In our study approximately 6% of patients were classified as belonging to emergency category ELISA1 and a similar proportion to ELISA5. This is similar to a previous study by the Federal Department of Public Health in Belgium.21 Most walk-in patients were categorised as ELISA3 and ELISA4; most use of resources, however, was found in the ELISA1 and ELISA2 patients. This reflects the considerable flexibility inherent in a broad scale, which in the case of ELISA does not come at the cost of increased difficulty or complexity. The high proportion of non-urgent patients was not surprising and is a widely described and complex issue, consideration of which is outside the scope of the present study. We feel that such apparent wasteful or inappropriate use of the ED is a strong argument in favour of the basic need for an accurate triage process.

Another important finding of our study is that our triage scale can reliably predict resource utilisation. In fact, our results demonstrated a strong correlation between resource consumption and triage category, although the scale was not designed for that perspective. This correlation has also been reported by others: −0.643 for FRENCH V.2, −0.54 for ESI V.3 and −0.48 for the the Canadian triage acuity scale.11 ,25

Most outcomes were also significantly correlated with ELISA: the higher the severity, the greater the number of hospitalisations, ICU admissions and deaths. This further confirms the strong validity of the scale.

As part of our interest in nursing triage, we are currently exploring the relevance of triage nurse-initiated medical investigations. In this way, we plan to study the appropriateness of x-ray, blood analysis and ECG ordering by triage nurses.

Our study contains some limitations. First, the data presented in this study came from one institution only, which may limit the ability to generalise our results to other facilities, because our hospital may have uncharacteristic patient characteristics that are not found in other units.

In the first phase of the study, written scenarios were used for the assessment of the reliability of the algorithm. The presentation of written scenarios is different from real cases and this could be a limitation; on the other hand, it would have been unethical to triage the same patient several times by several nurses.

Sixteen nurses participated on a voluntary basis in the study and we decided to make the ‘intrarater’ assessment by a group of 10 nurses, and the ‘interrater’ assessment also by a group of 10 nurses. This means some nurses participated in both assessments. Unfortunately, for practical reasons, only nine nurses were able to complete the ‘interrater’ assessment.

The relatively small number (n=34) of patients classified ELISA1 in the validity study is due to the fact that ELISA1 patients generally benefit from a pre-triage by prehospital emergency physicians and, therefore, were usually not included in the study. However, it should be noted that the number of triage1 patients in the validation study of ESI V.3 was 1% of 400,7 largely inferior to the number of patients included in our reliability study (6.25% of 554).

Finally, other physiological scoring methods such as medical early warning scoring were not considered and the benefit of the inclusion of vital signs against the use of symptoms-based discriminators only were not explored. Such a benefit could be evaluated in a further study, aiming at the implementation of the scale. However, ELISA remains a very easy-to-use scale.


In summary, the need for a specific, French-language triage tool in our ED led us to develop a new triage scale. The present study demonstrates that this scale is a valid triage tool with high interrater and intrarater agreement and shows considerable efficiency.


Supplementary materials

  • Supplementary Data

    This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

    Files in this Data Supplement:


  • Contributors All authors have read and approved the paper. They have met the criteria for authorship as established by the International Committee of Medical Journal Editors, believe that the paper represents honest work, and are able to verify the validity of the results reported.

  • Competing interests None.

  • Ethics approval The design of the study was approved by the hospital's human subject ethical review committee.

  • Provenance and peer review Not commissioned; externally peer reviewed.