Using a Modified 360° Multisource Feedback Model to Evaluate Surgery Residents in Shiraz University of Medical Sciences

INTRODUCTION:

Evaluation is one of the most important parts of every educational system. The correct use of it can boost the learners’ motivation for learner, motivation for learning and also provide us with useful feedback (1). Nowadays, it has been shown that traditional evaluation is not efficient enough to meet the theoretical and practical needs of the learners. As a result of one-way instructor-learner relationship in traditional learning, its evaluation used to be done merely based on the viewpoint of the faculty members (2). What’s more, it has been shown that these kinds of evaluations are neither efficient, nor sufficient in the assessment and maintenance of physicians ‘competence (3). Therefore, the need of new evaluation methods which are capable of doing multiple aspect assessment and providing feedbacks from learners has received worldwide attention (2). One of these methods which have been used in many fields, is the 360° or multisource feedback model of which the reliability has been reported as high as 90 percent in different studies.

More recently, during the two past decades, other authors have shown the priority of this method by evaluating the non-physician members communication skills, performance and humanity of physicians in training and practice (4,5, 6, 7).

In Semnan University, this 360-degree feedback model has been used for evaluating the academic performance of students who were admitted without university entrance exam (5).

 The object of MSF is to guide professional development. Studies show that practicing physician uses their feedback data to make changes in their practices (8).  In this method, a variety of evaluators such as patients, faculty members, medical colleagues, peers, coworkers and self-assessment completed a questionnaire to provide the physicians with feedback about a broad range of competencies (9,10). This method is being usually used for formative assessment and be used for the evaluation of actual behaviors to identified deficiencies (6,15).

It has been suggested that the physicians can use the feedback for their practice improvement (6). ACGME and The College of Physicians and Surgeons of Alberta (PAR program) have suggested this method for evaluating the required competencies; such as communication skill, interpersonal communication, professionalism, collaboration, collegiality ,clinical performance and patient care (6, 8). As a multi-aspect program, the residency programs are in need of an assessing method which can cover and assess the learners’ acquired competencies in different aspects. Having the mentioned characteristics, it seems that the 360° method is the best comprehensive tool. Although there are various methods used for assessing the residents, including standard questionnaire, Mini-CEX, DOPS, OSCE and standard patient, the common shortcoming in most of these methods is their limited scope of evaluation which does not exist in the 360° method (3,7, 8).

The validity, reliability and feasibility of this method in residents’ competency evaluations have been proven by many studies worldwide (7, 9, 10).

Skills and competencies in clinical practice are necessary for the medical profession. In fact, in patient care, doctors require practical skills in addition to scientific knowledge and the assessment of professionalism is still one of the difficult issues in medical education (19,20).

Likewise, Iran Health Ministry has also discussed the possibility of 360° model adaptation for residents’ evaluation; however, only limited pilot studies have actually evaluated the residents through this method in Iran and such an evaluation has never been done in Shiraz. Considering an increased demand for a high quality residency evaluation program and lack of an efficient instrument for doing so in Shiraz, we have decided to use this model to evaluate the present status of surgery residents ‘competencies in Shiraz University of Medical Sciences for the first time.

Methods

This cross-sectional and census study which assessed all 48 residents who were working in surgery departments of hospitals affiliated to Shiraz University of Medical Sciences.  The study group included 7 neurosurgeons, 18 general surgeons, 8 urologists and 15 orthopedists (45 men and 3 women in total) of which 24 were second-year(sophomore) and 24 were third-year (junior)residents. Informed consent was obtained from all the participants prior to study. This study was done based on a modified version of the 360° or multisource feedback model in which the patients were excluded by the assessors since most of them did not know how to complete the questionnaire and were uncooperative. The first-year residents who were unfamiliar with their specialty and senior residents who were not always present at rotations due to Board examinations were also excluded from the study. The items in each questionnaire were scaled using the 5 point Likert scale with an ascending 5 level of scaling (1=never to 5=always) and the choice “unable to assess” was considered for cases in which an individual could not answer the question.

The data was gathered using a valid and reliable Persian translation of the original 360° questionnaires which was developed in Calgary University (8, 14). The forward translation, which had exactly the same number of questions as the original questionnaire, was evaluated by a bilingual expert panel which confirmed its face validity. Then, it was back translated by an independent translator and its context validity was confirmed by an expert group. Its reliability was also confirmed using a pilot study and the Cronbach's alpha test.

The reliability of the questionnaires (Cronbach's α) were 87% for peers, 93% for self-assessment, 95% for faculty members and 97% for Co-worker questionnaires.

This modified 360°assessment model contained three different questionnaires: The Medical colleagues, Self-assessment and Co-worker. The first two examined professionalisms, communication skills, clinical performance and patient care. The Co-worker questionnaire focused on professionalism and communication skills. These questionnaires were distributed among four groups (The Co-workers (Nurses of operation room), the residents themselves, the faculty members and Peers).Using the experts’ opinions, three assessors were selected in each group. In other words, for each resident, a total number of 10 questionnaires were distributed, of which one was answered by the resident him/herself and three by each group (including the faculty members, Peers and Coworkers).This resulted in a four aspect evaluation.

Written informed consent was obtained from all the residents. The Medical colleague’s questionnaire included 34 items and was distributed among residents ‘peers and faculty members. The faculty members of each department were selected by educational deputy of that field and totally 12 faculty members were selected from 4 disciplines of study. The residents selected their peers themselves. The Self-assessment questionnaire had 34 items and was answered by all 48 residents themselves. The Co-worker questionnaire contained 19 items and was filled by nurses and department staffs who had worked closely with the resident during his / her rotation. In each group, the average score given to each question by the three assessors were measured and the mean total score for each questionnaire was calculated. The data was analyzed using the SPSS software version 17 and the statistical tests and models such as Pearson correlation coefficient, General Linear Model, T-test were used while they were needed.

RESULTS

The response rate were (95.8%), (95.8%), (95.1%) and (97.2%) in Self-assessment, Faculty members, Peers and Co-worker questionnaires, respectively. Less than 20% of the respondents selected the choice “unable to assess”. The reliabilities (Cronbach's α) of the questionnaires were 87% for peers, 93% for self-assessment, 95% for faculty members and 97% for Co-worker questionnaires.

Pearson correlation co-efficient was significant between the scores given by faculty members and peers (r=0.4, p=0.01) (Table 1), while no other statistically significant relationship were observed among the groups.

 Table 1:  Correlation matrix of total score of all of the raters.

* P-value < 0.05

 The highest mean score given by evaluators to residents belongs to the Coworkers group (Mean ±SD=149.6 ±17.23). Based on the General Linear Model (Repeated Measure), the differences between the mean scores given by the four groups of assessors were significant(P<0.05) (Table2).

Table 2: Trend of the scores obtained from the assessors

 Although it was not statistically significant, the nurses and medical staffs gave a better score to the second -years than third –year residents (P>0.05).

The third –year residents` Self-assessment mean scores were significantly higher in comparison with the second –years by using T-test (P value=0.01), although no significant differences were observed among the scores which were given to both groups by coworkers, peers and faculty members (Table 3).

Table 3: Comparison of the obtained scores in the second and third year residents.

DISCUSSION

The 360- degree evaluation method is a better assessment method because it includes perspective of assessor about full spectrum of the resident’s behavior.

In this study, through a modified 360° multisource feedback model, the second -year and third –year residents of general surgery, urology and orthopedics and neuro surgery competencies were evaluated. Not only this study was conducted for the first time in Shiraz, but also considering the acceptable Cronbach's α, it resulted in a reliable Persian translation of 360° multisource feedback questioners which provided us a perfect adapted instrument for our academic hospitals. The validity was obtained using the face validity which is similar to other studies and the same competencies (professionalism, communication skill and clinical performance and patient care were evaluated) (2, 3, 12,16). The results of this study suggest that the 360° multisource feedback model is a reliable, valid and feasible method for the residents’ evaluation; this result has confirmed the findings of other studies (3, 9,14). Considering the high voluntary participation and response rate, it seems that the residents found the implementation of this evaluation essential. This was consistent with other studies` findings (3, 9,15). Similar to other studies, our findings showed that using the 360° multisource feedback model is beneficial in the evaluation of the residents’ main competencies especially professionalism (9, 12, 13,16). In contrast with Joshi and Chandler reports, based on our findings, the residents who obtain higher grades and were clinically more skillful gave themselves higher scores) 11,17). This conflict is probably due to the training differences. Since Shiraz University of Medical Sciences’ hospitals are referral hospitals, each resident treat varied cases during their residency. They are also responsible for the supervision of many undergrads. These factors contributed to their higher self-confidence and in turn higher self-assessment. Our findings demonstrate that the third -year residents gave themselves higher scores in comparison with the second –years. This is probably due to the fact that they were more experienced and did more procedures. Therefore, they were more confident and used professionalism principles in their daily practices (21).

Unlike Massagli study, a significant relationship between the viewpoints of peers and faculty members were observed. This conflict might be because we used the same questionnaire for both groups since their aspect of evaluations were similar, as both groups had believed in the required clinical competences (12).

As other studies also stated, there was no significant relationship or very little correlation between other groups of the assessors (18).

Similar to other study, compared with the other evaluators, the Coworkers gave higher scores to the residents. (17). It shows that the Coworkers (nurses and staffs) mostly evaluate residents based on professionalism and communication skill (written or oral) and, while they can give good feedback regarding these general competences, more specific and clinical competences cannot be evaluated by them. These skills should be evaluated only by assessors with clinical insight such as faculties and peers.

This study was limited by the fact that it was performed on one filed (surgery), in a limited time period and the patients as an assessor was excluded. To enhance the psychometrics properties of the Persian translation of this questionnaire, re-conducting this study in other fields of specialty for a longer period of time is suggested.

Acknowledgement

The researcher appreciates all residents who provided us with their experiences.

Research committee approval and financial support:

This article is based on thesis done by Razieh Neshatavar as a  Partial Fulfillment of the Requirements for obtaining  a  Master Degree in  Medical Education and was approved in Shiraz University Medical Research Committee on April 9,2011  with approval number “ 90-5534 “

Conflict of interest

The author has no conflict of interest.