Article Text
PDF
Abstract
Objective To develop and validate a questionnaire to determine satisfaction with the hospital emergency department.
Design Cross-sectional study to validate a telephone questionnaire designed in Spanish by a panel of experts.
Setting The emergency department of Hospital Gregorio Marañón, a tertiary level hospital of the Spanish national health system.
Participants A sample of 651 emergency department patients completed the questionnaire.
Main Outcome Measures The psychometric properties of the questionnaire were evaluated; namely, construct, criterion validity, predictive validity and internal consistency.
Results Two dimensions—comfortable service and personalised service—were identified from the exploratory factor analysis, and these accounted for 63% of the variance. Both factors showed a positive correlation with the global assessment items ‘global satisfaction with the attention received in the emergency ward’ and ‘Would you recommend this emergency department?’ The predictive validity of the questionnaire was assessed by means of discriminant analysis, which showed that 66.7% of patients were correctly classified. Internal consistency measured by Cronbach's alpha resulted in a value greater than 0.80 for both dimensions.
Conclusions This questionnaire fulfils the necessary psychometric properties to be considered a useful and reliable tool for measuring patient satisfaction with hospital emergency services.
- Management
- medical emergencies
- patient satisfaction
- quality assurance
- quality of health care
- questionnaire
- reproducibility of results
Statistics from Altmetric.com
- Management
- medical emergencies
- patient satisfaction
- quality assurance
- quality of health care
- questionnaire
- reproducibility of results
The quality of health care has traditionally been evaluated from the perspective of health professionals and measured using direct and indirect outcomes of medical care. However, not only do patients assess quality in terms of the results of medical care, they also value their perception of the care provided. Therefore, to ensure adequate quality of healthcare provision, it is necessary to satisfy patient needs and expectations. Patient-established criteria are fundamental for effective quality assessment,1 2 and patient perceptions are an essential component and measure of evaluation of and improvement in health care.3
Several methods are available, including complaints and suggestions analysis4 and qualitative research techniques,5 although quantitative research methods are the most commonly used, mainly in the form of satisfaction surveys or questionnaires.6 7
Not all questionnaires fulfil the psychometric properties that guarantee validity and reliability, which are prerequisites for results to be extrapolated to the general population. The questionnaire must be reliable (assessed in terms of internal consistency, temporal stability and interobserver agreement), accurate and change sensitive. Validating a questionnaire involves ensuring that it fulfils construct, content, criterion validity and reliability criteria, through repeated applications and contrasted measurements.8
The objective of this study was to develop and validate a questionnaire that would serve to evaluate the satisfaction of patients attending a hospital emergency department.
Materials and methods
Study design
We undertook an observational, cross-sectional telephone survey to validate satisfaction with hospital emergency care.
Study setting and population
Our setting was the emergency department of Hospital Gregorio Marañón (Madrid, Spain), a tertiary care national health system hospital with a mean of 500 visits per day. The patient is admitted in 11% of cases.
The target population was composed of patients over 16 years of age who sought medical attention in the emergency room. Those with signs of stupor or coma and the institutionalised elderly were excluded. Sample size was calculated to achieve 5% accuracy, a 95% CI and an expected response rate of 70% (average response rate based on the experience of our hospital). A total of 651 surveys was required.
Sample size was calculated taking into account the following strata:
Patients attending the emergency department (two strata):
Patients attending the emergency department between 08:00 and 22:00 hours (80%).
Patients attending the emergency department between 22:00 and 08:00 hours (20%).
Day of the week on which the patient comes to the emergency department (two strata):
Patients attending the emergency department daily (Monday to Friday) (80%).
Patients attending the emergency department on the weekend (20%).
Participants were selected within each stratum at random from the list of patients who attended the emergency room. Participants who could not be located or did not participate in the study were replaced by another participant randomly selected from the same stratum.
Study protocol
The questionnaire was developed by an expert panel consisting of staff from different departments of the hospital: emergency; administrative patient care services; and preventive medicine and quality management. Experts in sociological studies from the autonomous community of Madrid quality service and citizen enquiry office supported and collaborated with the expert panel.
Group members chose a list of key issues to evaluate, based on their own experience as a patient and as a health professional. In parallel, a literature review was conducted to identify predictive factors of satisfaction in patients who attend the emergency department.9–12
The information collected for the study was treated according to the provisions of Spanish Law 15/13 December 1999 on the protection of individual patient data. Before the start of the study, appropriate approval was obtained from the authorities that regulate the application of quantitative techniques and qualitative social research to measure the quality of services provided by Madrid local government organisations. Similarly, when the patient answered the phone, we explained the questionnaire and its purpose, and respected their decision to participate or not.
To control possible interviewer bias, a pilot study on patients treated in the emergency department was performed to identify the most appropriate type of question, whether the wording was understandable and the length of the question was appropriate, whether the categorisation of the answers was correct, whether there were questions the interviewee refused to answer, and whether the length of the questionnaire was acceptable to the interviewee.
Measurements
The Delphi consensus technique was applied to the resulting preliminary 30-item list, with the input of seven participating experts including medical and nursing staff from the emergency department, patient administrative care service professionals, experts in healthcare quality and patients. Two Delphi rounds were applied to the questionnaire, which was called ‘satisfacción del usuario con el área de urgencia hospitalaria’ (SUAUH; user satisfaction with the hospital emergency department) and contained 13 items exploring satisfaction-related aspects, each scoring in five categorically ordered responses from 1 (very good) to 5 (very bad). The response scale, graded in five categories, was based on guidelines reported elsewhere,13 14 the personal experience of the research group and the range of responses given by patients.
In addition to the questions mentioned above, five questions exploring sociodemographic issues and emergency service use were added, as were 10 questions to be used as reference criteria. A final question was asked to obtain a global evaluation of the services received in the emergency department on a scale from 0 (very bad) to 10 (very good). The questionnaire was divided into four different domains: access and admission; information and personal manner of health staff; physical structure and catering and discharge and global evaluation. This division was introduced in order to track the logical sequence of use of an emergency department.
The questionnaire was sent to a second expert panel (five experts including emergency ward professionals and quality experts) to establish content validity. It was then piloted with emergency room patients to assess their degree of understanding and acceptance. The workload and time required for the interview were also assessed. Observations and suggestions were considered and, after making changes in the formulation of some questions, a final version of the questionnaire was drawn up. Interviewers were trained to implement the questionnaire as efficiently as possible.
Data analysis
A descriptive analysis for quantitative variables was carried out using the t test and an analysis of variance for comparison of the means. Categorical variables were described as percentages and proportions using the χ2 test.
Construct validity was evaluated using principal component exploratory factor analysis15 followed by a varimax rotation. Assumptions for this method were confirmed by the Kaiser–Meyer–Olkin (KMO) sample adequacy measurement, the Bartlett test of sphericity and the correlation matrix determinant.
Criterion validity was evaluated based on correlation of the questionnaire with two items used as global assessment variables: ‘global satisfaction with the attention received in the emergency ward’ and ‘Would you recommend this emergency department?’ Discriminant analysis was used to assess predictive validity16 in order to estimate the percentage of patients correctly classified by the questionnaire in the groups previously defined by the aforementioned reference criteria. Cronbach's alpha was used to evaluate the internal consistency of the questionnaire as a whole and for each of the independent factors.17
Results
We telephoned 651 patients (with a pre-established maximum of four attempts per patient) of whom 296 answered the phone and completed the interview (45.5% response rate). Non-response was due to the following:
None of the four calls answered (57.6%)
The phone number did not exist or did not correspond with the patient's information (34.3%)
The user refused to participate in the study (8.1%).
Table 1 shows the sociodemographic features of the patients who replied to the questionnaire—there were no differences between patients who answered and those who did not, except for gender (women answered more than men, p<0.01).
Sociodemographic features of patients who participate
Questionnaire validity
Principal component exploratory factor analysis identified three factors that accounted for 65% of variance: one factor was responsible for 37% of variance and two less important factors accounted for 19% and 9%, respectively. Assumptions for the use of this method were previously checked: KMO 0.91, Bartlett sphericity test p<0.05 and correlation matrix determinant p<0.001. The third component of the model appeared as a poorly defined factor—it showed a high correlation with only one variable (ease of administrative procedure) and accounted for a small percentage of variance.
Therefore, we performed a second factor analysis after eliminating two variables (ease of administrative procedures and emergency ward cleanliness) that were not appropriately explained by the three-factor model. A varimax rotation combined with a Kaiser normalisation revealed a factor solution with two factors accounting for 63% of variance (KMO 0.90, Bartlett sphericity test p<0.05, correlation matrix determinant p<0.001). The first factor, personalised service, contains six items and accounts for 33% of variance. It includes issues related to kindness, personal manner and professional competence. The second factor, comfortable service, accounts for 30% of variance and contains five items related to coordination and organisation, information received and comfort of facilities. Table 2 shows the results of the factor analysis.
Results of the factor analysis and factorial saturation of items
Criterion validity was analysed by taking two reference criteria into consideration: ‘global satisfaction with the attention received in the emergency ward’ and ‘Would you recommend this emergency department?’ The non-parametric Spearman rank correlation coefficient was positive between the first reference criterion and the two identified factors, with a correlation coefficient of 0.35 (95% CI 0.25 to 0.45) for personalised service and 0.59 (95% CI 0.51 to 0.66) for comfortable service. As for the second reference criterion, the validity analysis using logistic regression obtained an OR of 2.31 (95% CI 1.55 to 3.47) for personalised service and 4.21 (95% CI 2.67 to 6.64) for comfortable service.
The discriminating analysis carried out using the variable ‘global satisfaction with attention received in the emergency ward’ as reference criterion confirmed 66.7% of patients (N=156) as being correctly classified, a percentage that rises to 92.1% when the reference criterion used is ‘Would you recommend this emergency department?’
Questionnaire consistency
Cronbach's alpha for the original 13-item questionnaire was 0.90 (N=158). Item–total correlation values varied between 0.44 and 0.78, excluding values for the items ease of administrative procedures (0.31) and emergency ward cleanliness (0.37). When the latter were eliminated, Cronbach's alpha was calculated for the resulting 11-item questionnaire, resulting in 0.92 (N=166) and not improving with the exclusion of any other variable. Item–total correlation values varied between 0.47 and 0.81. Cronbach's alpha for personalised service was 0.90 (N=251), with item–total correlation values varying between 0.66 and 0.79. Cronbach's alpha for comfortable service was 0.83 (N=182), with item–total correlation values between 0.56 and 0.71 (table 3).
Item–total correlation for each factor variables personalised service and comfortable service and Cronbach's alpha after variable exclusion
Final version
After the validation process, we obtained an 11-question version of the questionnaire (SUAUH-11, appendix I) to assess patient satisfaction with hospital emergency care.
Discussion
Many tools have been developed to measure patient satisfaction, although the questionnaire or survey is the most widely used. It was not possible to find a questionnaire validated in our setting that included the aspects we considered necessary to evaluate the satisfaction of patients treated in the hospital emergency department. This was probably due to the heterogeneous nature of the services provided by the emergency department and the features of a tertiary, highly specialised institution serving a large number of patients from the autonomous community of Madrid and other Spanish autonomous communities. Therefore, we designed and developed a specific questionnaire.
Validation of the questionnaire using principal components analysis initially identified a structure consisting of three factors accounting for 65% of the variance, with one main factor and two minor factors. We performed a second factor analysis in which the variables ‘ease of administrative procedures’ and ‘cleanliness of the emergency room’ were eliminated. From the clinical point of view, removing these two variables from the questionnaire does not seem to have much relevance. Administrative procedures are not usually performed by the patient. Their families or carers are usually responsible for this part of the visit—with the result that the patient is often unaware of the ease or difficulty of administrative procedures. Cleaning of the emergency room is performed by a subcontractor whose work is reviewed regularly; consequently, their results are evaluated regularly and are not usually a cause for complaint by patients. In addition, the time the patient remains in the emergency department is limited, so cleanliness is rarely an important aspect of their stay. After eliminating these two variables, two factors accounting for 63% of the variance were found. After analysing the two factor solutions, we selected the second factor solution, as this model had more stability at the expense of minimal loss in percentage of variance explained. The first factor was ‘personalised service’, and incorporates professional competence, kindness and amiability, aspects perceived subjectively by the patient. The second factor called ‘comfortable service’ refers to the organisational objectives of the emergency department. Analysis of the validity of the criteria revealed that both factors were positively correlated with the overall assessment of the visit.
The internal consistency of the questionnaire, initially assessed using the Cronbach's alpha for the whole questionnaire without removing the variables of ‘ease of administrative procedures’ and ‘cleanliness of the emergency room’, gave an alpha value of 0.90. When the item–total correlations were analysed, these two variables showed a lower correlation. When the two variables were eliminated, the Cronbach's alpha value was 0.90 for ‘personalised service’ and 0.83 for ‘comfortable service’; however, the value did not improve when variables were excluded. The item–total correlation was adequate in both factors.
Patients evaluated our institution's emergency department positively, with a score of 7 (on a scale of 1–10). The positive reviews often obtained in satisfaction surveys18 can be explained by the fact that diagnosis and treatment are often immediate. In our study, 55.1% of patients considered that their health problem improved somewhat or completely after attending the emergency department. Moreover, we must take account of the bias arising from the feeling of gratitude and satisfaction the respondents feel on being asked to provide their opinion in a survey.
Treatment and the professional competence of health professionals are the most valued aspects; in addition, professional competence was one of the aspects that most affect the global satisfaction index. Some of these items, such as competence of health professionals, are difficult for patients to evaluate; however, our intention was not to measure these items, but to obtain patients' perception of them.
Improving the comfort of the emergency room would lead to a substantial increase in satisfaction. Patients who attend the emergency department for the first time have a poorer opinion of the service than those who have already attended on other occasions. This could be because their expectations are higher before they actually visit the service.
Time spent in the emergency department was considered appropriate by 52.5% of those surveyed; however, most interviewees suggested reducing waiting time. This finding is consistent with other surveys to determine satisfaction with the emergency department.
Limitations
A possible limitation is that the patients did not participate in the design of the questionnaire. However, this is partly remedied by the participation of professionals from the hospital's patient care services, who have a broad knowledge of patients' complaints and suggestions. Furthermore, in the literature review to identify factors influencing patient satisfaction or dissatisfaction, special attention was paid to studies on qualitative research techniques based on focus groups, nominal groups, or interviews.
Our response rate was 45.5%, somewhat lower than that found in other studies using the same methodology.18 19 Although calls were made in the morning and evening (until 18:00 hours) to try to locate as many participants as possible, 92% of non-response was due to the inability to locate participants after four attempts.
To minimise bias, interviewers were trained to standardise questions as much as possible, and to ensure that the questions were well understood by the respondent. To reduce recall bias, interviews were held no more than 60 days after the visit to the emergency department.
Conclusions
The results allow us to recommend this questionnaire for quick and easy assessment of patient satisfaction with the hospital emergency department with a high level of reliability and validity. Similarly, large-scale administration of this questionnaire in the next few years will allow us to evaluate its sensitivity and to check whether the information provided ultimately leads to an improvement in the services offered by the emergency department. Improving and adapting the facilities and transmission to users can significantly improve the view that users have of the service.
Acknowledgments
The authors would like to thank Amparo Mancebo Izco, Head of the Sociological Studies Department at the Madrid Quality of Service and Citizen Enquiry Office, José María Bellón Cano, statistical expert at the Gregorio Marañón Hospital Department of Preventive Medicine and Quality Management, Thomas O'Boyle and all the department's administrative staff for their invaluable help in this study and their collaboration in data processing tasks.
Appendix I SUAUH-11 (Cuestionario de Satisfacción del Usuario con el Área de Urgencia Hospitalaria, User Satisfaction with Hospital Emergency Despartment Questionnaire)
Access and admission
1. How would you describe the personal manner of the admission staff?
Very good [] Good [] Average [] Bad [] Very bad []
Information and personal manner
2. How would you describe the personal manner of the medical staff?
Very good [] Good [] Average [] Bad [] Very bad []
3. How would you describe the professional competence of the medical staff?
Very good [] Good [] Average [] Bad [] Very bad []
4. How would you describe the information provided by medical staff about your health problem?
Very good [] Good [] Average [] Bad [] Very bad []
5. How would you describe the personal manner of the nursing staff?
Very good [] Good [] Average [] Bad [] Very bad []
6. How would you describe the professional competence of the nursing staff?
Very good [] Good [] Average [] Bad [] Very bad []
7. How would you describe the peace of mind and support provided by medical and nursing staff?
Very good [] Good [] Average [] Bad [] Very bad []
Physical structure and catering
8. How would you describe the comfort of the emergency ward?
Very good [] Good [] Average [] Bad [] Very bad []
Global evaluation
9. How would you describe the organisation and coordination of the emergency department?
Very good [] Good [] Average [] Bad [] Very bad []
10. How would you describe the information provided by information staff?
Very good [] Good [] Average [] Bad [] Very bad []
11. In general, how would you describe your stay in the emergency ward?
Much better than I expected [] Better than I expected [] As I expected [] Worse than I expected [] Much worse than I expected []
References
Footnotes
Competing interests None to declare.
Patient consent Obtained.
Ethics approval This study was conducted with the approval of the Area Evaluation of Public Services of the Directorate General of Quality of Services and Citizens a mandatory report on the design to be applied in the study as established by decree 149/2004 of 21 December, the governing council, which establishes the organisational structure of the presidency and council of the order of 18 November 2004 and Deputy Minister of Justice and Interior, which regulates the use of quantitative techniques and qualitative social research to measure the quality of services provided by Madrid.
Provenance and peer review Not commissioned; externally peer reviewed.