Evaluating the Anesthesiology Residents’ Performance, Using a Modified 360-degree Assessment Questionnaire in Shiraz University of Medical Sciences

Introduction:

Evaluation is one of the most important parts of every educational system. Its desirable use can improve the learners’ motivation and causes beneficial feedback for learning (1).

The Accreditation Council for Graduate Medical Education (ACGM) initiated the outcome project to increase the emphasis on educational outcomes in the accreditation of residency programs and confirmed 6 general competencies for residents:

1. Medical knowledge 2. Patient care 3. Professionalism 4. Practice-based learning and improvement 5. Systems-based practice 6. Interpersonal and communication skills (2).

In recent decades, worldwide attentions were increased in many countries for example North America and Europe to evaluate physician’s performance because Professional behavior in medicine has been affected by teaching and evaluation professionalism. The assessment of residency programs have gained prominence and become a necessity (3, 4).

Among the assessment methods, 360-degree feedback or multisource feedback is the most common, comprehensive, and efficient (5, 6).

In traditional assessment methods, assessing qualifications such as professionalism, communication skills, and interpersonal skills was made by professors. In such approaches, when assessing a physician’s performance, the focus was usually on her/his clinical rather than behavioral qualifications. However, obtaining other viewpoints alongside the professors’ assessment can provide a more complete picture of the residents’ performance in different situations (7, 8).

The medical school at Shiraz University of Medical Sciences with over 50 years background has emphasized the use of 360-degree assessment in its planning as a new assessment tool.

The purpose of this study was to translate and determine the validity and reliability of the Persian version of the 360-degree assessment for anesthesiology residents. This questionnaire consists of different domains to measure the general capabilities including communication and interpersonal skills, professionalism and residents’ clinical care skills from the viewpoints of residents themselves, faculty members, nurses (operation room staff, anesthetic technicians, and recovery room nurses), colleagues, and patients. This is done for the first time in Iran and can be used as a main method of new assessments for measuring general competencies of residents.

Methods:

In this cross-sectional and census study, there were 35 second-year and third-year residents (15 and 20, respectively). The questionnaire developed by Calgary University in Canada was used in this study to investigate the psychometric features (reliability, validity, and variance analysis). This evaluation was done based on a modified version of the 360° or multisource feedback model. For each resident, five raters were selected. The questionnaire was translated into Persian and then into English. The back-translated English version was cross-matched with the original questionnaire. A bilingual panel of experts experienced in translation and development of questionnaires checked the translated version of the questionnaire and reached a consensus regarding the clarity and accuracy of the items in the questionnaire.

Three main aspects of validity considered in the study were face validity, content validity and construct validity. For reliability, Cronbach's alpha coefficient was used. The purpose of the project was explained to the residents and they completed the informed consent from prior to study.

All second and third year residents who were actively engaged in anesthetic induction and were in close contact with their professors were chosen. The first and fourth year residents were excluded from the study, the former due to lack of familiarity and the latter because of being involved in non-operation room services.

The raters included five groups of faculty members, operation room staff (senior anesthetic technicians and recovery room nurses), residents’ colleagues, patients and residents themselves (self-assessment). The inclusion criterion was based on being in the operation or recovery room with the residents at least three months in a year and frequently working with the residents.

The residents were rated via four questionnaires: the questionnaire for nurses with 19 items, the questionnaire for faculty members and colleagues with 29 items, the questionnaire for patients with 11 items, and the self-assessment questionnaire with 29 items.

The domains to be assessed were the residents’ communication skills, clinical care and professionalism. Each item was measured on a 5-point Likert scale with an ascending 5 level of scaling (1=never to 5=always) and the choice “unable to assess” was considered for items in which an individual could not answer the question.

For data collection, we referred to the operation rooms in Namazi and Khalili Hospitals when the residents were on-call shifts. The questionnaires were distributed among the raters and they were explained how to complete them. The data was analyzed using the SPSS software version 17 and the statistical tests and models such as Pearson correlation coefficient, General Linear Model, and T-test.

Results:

Thirty one residents (88%) took part in the study. Cronbach's alpha coefficient for each questionnaire was over 0.80 (Table 1). Regarding the construct validity, the correlation between the items constituting each domain and the domain itself was over 0.40. The items of the questionnaires were divided into 3 domains: Interpersonal and communication skills, Clinical care, and Professionalism.

Table 1. Mean, Standard deviation, and Cronbach's alpha for each Questionnaire

Pearson correlations were determined among the domains for each rater. Regarding professionalism, a statistically significant difference was found between the colleagues and patients’ viewpoints (r=0.21). Considering clinical care, there was also found a statistically significant difference between the faculty members and patients’ viewpoints (r = 0.2) (Tables 2 & 3). Regarding the communication skills, no statistically significant difference was found between the raters’ viewpoints.

 Table 2. Correlation coefficient among raters for professionalism

 Table 3. Correlation coefficient among raters for clinical care

To find out the mean difference among all raters, this study used repeated measures. The mean difference in each domain showed that the difference among the four groups of raters was statistically significant (P < 0.5) (Table 4).

Table 4. Comparison of the means among raters for each domain

The comparison of the means revealed some differences among the raters’ viewpoints (Table 4). Self-raters gave the highest scores to themselves, followed by the colleagues, faculty members, nurses and finally the patients who gave the lowest scores.

 Discussion:

The 360- degree evaluation method is a useful assessment method because it contains assessors’ point of view about multi aspects of the residents’ behavior (1).

Health providers’ accountability, patient care, etc. have prompted some developments in health education in recent years and different evaluation methods have been proposed to assess them.

In this study, the residents themselves (self-assessment), colleagues, nurses, faculty members and patients evaluated the anesthesiology residents for the first time in Iran, using 360-evaluation instrument. To interpret the residents’ performance better, items of the questionnaire were divided into different domains. This evaluation method was started in 1980, being widely used in some organizations and industries. In the medical field, the Accreditation Council for Graduate Medical Education (ACME) from 1999 and the Physician Achievement Review (PAR) by Calgary College of Physicians and Surgeons from 1996 used 360 degrees assessment as a formative performance assessment to provide structured multi-source feedback of physicians’ performance. Nowadays it is more used to evaluate such providers as family physicians, surgeons, anesthetists and internists (9).

ACME developed the tool to assess the residents’ performance in 6 general domains including patient care, medical knowledge, professionalism, communication skills, task-based learning, and system-based performance. Regarding the medical field, 360degree assessment is particularly suitable to assess communication skills, altruism, esprit de corps, and professionalism and it is currently used in North America and Europe (10,11).

Reliability, validity, and feasibility of a 360-degree evaluation were used for radiology residents (12). Otherwise, Anesthesiology residents must possess a high competency of knowledge and skills in order to assess, treat and impart the required information to the patient for their life quality promotion (13). Different studies have shown that 360-degree assessment is an effective and efficient tool to provide feedback in both clinical and nonclinical practices. This tool can assess the given individual retrospectively, synchronously, and separately by people working in the same environment (10, 13).

Our study showed that the reliability of the questionnaire was over 0.80. This confirms the validity, reliability and accessibility of the questionnaire and is in accordance with the results of other studies (12, 14). As a result, this tool has the capacity to be used in hospitals and its Persian translation has the same capability as its original form because of its high reliability and validity.

Regarding the domain of communication skills, the correlations between the rates’ viewpoints were not statistically meaningful. This might be due to the limited number of items in the questionnaire which did not cover all aspects of the residents’ competence in this domain. In a study on surgery residents, the relationship between the viewpoints of the faculty members and the residents’ colleagues were statistically significant in this domain. It seems that in 360-degree assessment, different raters rate the study subjects differently. This is mainly due to the manner of interaction and familiarity among the raters and the study subjects (14).

A study in China showed that there was a statistically significant difference between the raters’ viewpoints and regarding the professionalism domain, there was a relationship between the patients’ viewpoints and those of the faculty members (2). In the present study, a relationship was found between the patients’ viewpoints and those of the colleagues. It can be concluded that the colleagues and patients have a key role in the professionalism assessment of the anesthesiology residents.  

The low scores given by the faculty members indicate that they put more emphasis on the development of professionalism, clinical care and communication skills. The high scores by residents themselves (self-assessment) and its significant difference with other raters’ scores are in compliance with the findings of other studies. This reveals the fact that self-assessment does not enjoy high reliability. The study by Lockyer, et al. on anesthesiology graduates showed that the scores in self-assessment were more than those by colleagues (15).

A study at the University of Wisconsin-Madison and a study by Meng, et al. at Petersburg University on radiology and anesthesiology residents, using 360-degree assessment, showed that this tool was reliable and valid enough to measure professionalism and communication skills (12, 13).

Our study shows that the 360- degree instrument, with ratings by faculty, nurses, medical students, and patients’ opinion, can be used to evaluate the competencies of residents during training, which accords with the results of other study (16).   

A study in Boston by Lagoo J showed that Surgeon behavior, as assessed by 360-degree review, was associated with malpractice claims. These findings highlight the importance of teamwork and communication in exposure to malpractice. Although the nature of malpractice claims is complex and multifactorial, the identification and modification of negative physician behaviors may mitigate malpractice risk and ultimately result in the improved quality of patient care (17).

Another study was made by Jani H in India showed that Regular orientation programs for professionalism with 360 degree evaluation and subsequent feedback to the resident doctor about their strength and weaknesses could definitely bring out behavioral change in the resident doctor in practice (18).

A study by Berger JS showed that Compared to receiving traditional feedback from faculty-only, residents improved their performance in Interpersonal and Communication Skills after first receiving 360-degree feedback. This method of feedback may also facilitate developing the competency of Practice-Based Learning and Improvement although educational studies with larger sample sizes are needed to confirm the observed trends (19).

The present study showed that the Persian version of 360-degree scale is a practical and effective assessment tool with proper reliability and validity to measure the residents’ competence. Despite the fact that this questionnaire is based on a standard questionnaire and has the capacity to assess residents’ competence, it is suggested to be applied in other specialties to get more definite results. Moreover, the items in the questionnaire should be constructed so that all aspects of the domain will be covered. Finally, regarding the nature of each discipline, different values should be given to different raters’ scores, i.e. the assessments of all raters should not be considered equally important.

Ethical considerations 

Ethical issues (Including plagiarism, informed consent, misconduct, data fabrication and/or falsification, double publication and/or submission, redundancy, etc.) have been completely observed by the authors.

Acknowledgments:

This article is based on proposal research which was approved in Shiraz University Medical Research Committee on December 7, 2013. We thank Vice Chancellor of research who provided funding.

Financial Support: The present study was financially supported by the Vice Chancellor of Research at Shiraz University of Medical Sciences with approval number: 92-6796.

Conflict of interest: The authors report no conflicts of interest concerning the material and method used in this study