Using a Modified 360° Multisource Feedback Model to Evaluate Surgery Residents in Shiraz University of Medical Sciences

Document Type: Original Article

Authors

1 Quality Improvement in Clinical Education Research Center, Education Developmental Office, Medical School, Shiraz University of Medical Sciences, Iran

2 Quality Improvement in Clinical Education Research Center, Education Developmental Center, Shiraz University of Medical Sciences, Iran

3 Medical Ethics Department, Shiraz Medical School, Shiraz University of Medical Sciences, Shiraz ,Iran

4 Department of Biostatistics, Infertility Research Center, Shiraz University of Medical Sciences, Shiraz ,Iran

Abstract

Background: Traditionally, the residents ‘competencies such as professionalism, interpersonal and communication skill have been assessed merely based on their faculty’ point of view without feedback. These assessments were neither efficiently, nor sufficiently cover the multi aspects` of residents ‘competencies. These multi aspects competences mandate the need for more all-around evaluations such as 360° degree. The purpose of the present study was to apply the 360° multisource feedback model for assessing the surgery residents` competencies of Shiraz university of Medical Sciences.
Methods: This cross sectional study was performed on 48 residents of the second and third year who were working in the surgery departments. The data were gathered using translated questioners on a 5-point likert scale and distributed to 3 raters (nurses, faculty members and peers). The reliability and validity of the questionnaire were approved by some experts. The data was analyzed by Pearson correlation Coefficient, General Linear Model, T- test and SPSS (17) software.
Results: The Internal Consistency reliabilities were up to 80% for all questionnaire. The Pearson correlation co-efficient was significant between the scores given by faculty members and peers (r=0.3, p=0.01). The highest mean score given by the evaluators to residents belonged to the Coworker group (149.6, ±17.23). The third-year residents gave themselves significantly higher scores in comparison with the second year residents (P value=0.01).
Conclusions: High Cronbach’s α demonstrates that the translated questioners were reliable. It seems that based on the valid and reliable results of this study, the 360° multisource feedback model is a good frame work for surgery resident’s assessment in Iran.

Keywords


INTRODUCTION:

Evaluation is one of the most important parts of every educational system. The correct use of it can boost the learners’ motivation for learner, motivation for learning and also provide us with useful feedback (1). Nowadays, it has been shown that traditional evaluation is not efficient enough to meet the theoretical and practical needs of the learners. As a result of one-way instructor-learner relationship in traditional learning, its evaluation used to be done merely based on the viewpoint of the faculty members (2). What’s more, it has been shown that these kinds of evaluations are neither efficient, nor sufficient in the assessment and maintenance of physicians ‘competence (3). Therefore, the need of new evaluation methods which are capable of doing multiple aspect assessment and providing feedbacks from learners has received worldwide attention (2). One of these methods which have been used in many fields, is the 360° or multisource feedback model of which the reliability has been reported as high as 90 percent in different studies.

More recently, during the two past decades, other authors have shown the priority of this method by evaluating the non-physician members communication skills, performance and humanity of physicians in training and practice (4,5, 6, 7).

In Semnan University, this 360-degree feedback model has been used for evaluating the academic performance of students who were admitted without university entrance exam (5).

 The object of MSF is to guide professional development. Studies show that practicing physician uses their feedback data to make changes in their practices (8).  In this method, a variety of evaluators such as patients, faculty members, medical colleagues, peers, coworkers and self-assessment completed a questionnaire to provide the physicians with feedback about a broad range of competencies (9,10). This method is being usually used for formative assessment and be used for the evaluation of actual behaviors to identified deficiencies (6,15).

It has been suggested that the physicians can use the feedback for their practice improvement (6). ACGME and The College of Physicians and Surgeons of Alberta (PAR program) have suggested this method for evaluating the required competencies; such as communication skill, interpersonal communication, professionalism, collaboration, collegiality ,clinical performance and patient care (6, 8). As a multi-aspect program, the residency programs are in need of an assessing method which can cover and assess the learners’ acquired competencies in different aspects. Having the mentioned characteristics, it seems that the 360° method is the best comprehensive tool. Although there are various methods used for assessing the residents, including standard questionnaire, Mini-CEX, DOPS, OSCE and standard patient, the common shortcoming in most of these methods is their limited scope of evaluation which does not exist in the 360° method (3,7, 8).

The validity, reliability and feasibility of this method in residents’ competency evaluations have been proven by many studies worldwide (7, 9, 10).

Skills and competencies in clinical practice are necessary for the medical profession. In fact, in patient care, doctors require practical skills in addition to scientific knowledge and the assessment of professionalism is still one of the difficult issues in medical education (19,20).

Likewise, Iran Health Ministry has also discussed the possibility of 360° model adaptation for residents’ evaluation; however, only limited pilot studies have actually evaluated the residents through this method in Iran and such an evaluation has never been done in Shiraz. Considering an increased demand for a high quality residency evaluation program and lack of an efficient instrument for doing so in Shiraz, we have decided to use this model to evaluate the present status of surgery residents ‘competencies in Shiraz University of Medical Sciences for the first time.

Methods

This cross-sectional and census study which assessed all 48 residents who were working in surgery departments of hospitals affiliated to Shiraz University of Medical Sciences.  The study group included 7 neurosurgeons, 18 general surgeons, 8 urologists and 15 orthopedists (45 men and 3 women in total) of which 24 were second-year(sophomore) and 24 were third-year (junior)residents. Informed consent was obtained from all the participants prior to study. This study was done based on a modified version of the 360° or multisource feedback model in which the patients were excluded by the assessors since most of them did not know how to complete the questionnaire and were uncooperative. The first-year residents who were unfamiliar with their specialty and senior residents who were not always present at rotations due to Board examinations were also excluded from the study. The items in each questionnaire were scaled using the 5 point Likert scale with an ascending 5 level of scaling (1=never to 5=always) and the choice “unable to assess” was considered for cases in which an individual could not answer the question.

The data was gathered using a valid and reliable Persian translation of the original 360° questionnaires which was developed in Calgary University (8, 14). The forward translation, which had exactly the same number of questions as the original questionnaire, was evaluated by a bilingual expert panel which confirmed its face validity. Then, it was back translated by an independent translator and its context validity was confirmed by an expert group. Its reliability was also confirmed using a pilot study and the Cronbach's alpha test.

The reliability of the questionnaires (Cronbach's α) were 87% for peers, 93% for self-assessment, 95% for faculty members and 97% for Co-worker questionnaires.

This modified 360°assessment model contained three different questionnaires: The Medical colleagues, Self-assessment and Co-worker. The first two examined professionalisms, communication skills, clinical performance and patient care. The Co-worker questionnaire focused on professionalism and communication skills. These questionnaires were distributed among four groups (The Co-workers (Nurses of operation room), the residents themselves, the faculty members and Peers).Using the experts’ opinions, three assessors were selected in each group. In other words, for each resident, a total number of 10 questionnaires were distributed, of which one was answered by the resident him/herself and three by each group (including the faculty members, Peers and Coworkers).This resulted in a four aspect evaluation.

Written informed consent was obtained from all the residents. The Medical colleague’s questionnaire included 34 items and was distributed among residents ‘peers and faculty members. The faculty members of each department were selected by educational deputy of that field and totally 12 faculty members were selected from 4 disciplines of study. The residents selected their peers themselves. The Self-assessment questionnaire had 34 items and was answered by all 48 residents themselves. The Co-worker questionnaire contained 19 items and was filled by nurses and department staffs who had worked closely with the resident during his / her rotation. In each group, the average score given to each question by the three assessors were measured and the mean total score for each questionnaire was calculated. The data was analyzed using the SPSS software version 17 and the statistical tests and models such as Pearson correlation coefficient, General Linear Model, T-test were used while they were needed.

 

RESULTS

The response rate were (95.8%), (95.8%), (95.1%) and (97.2%) in Self-assessment, Faculty members, Peers and Co-worker questionnaires, respectively. Less than 20% of the respondents selected the choice “unable to assess”. The reliabilities (Cronbach's α) of the questionnaires were 87% for peers, 93% for self-assessment, 95% for faculty members and 97% for Co-worker questionnaires.

Pearson correlation co-efficient was significant between the scores given by faculty members and peers (r=0.4, p=0.01) (Table 1), while no other statistically significant relationship were observed among the groups.

 Table 1:  Correlation matrix of total score of all of the raters.

 

Assessor

 

Self-assessment

peer

faculty member

nurse

Self-assessment

-

-

-

-

Peer

0.1

-

-

-

Faculty member

0.04

0.4*

-

-

Nurse

0.2

0.07

0.2

-

* P-value < 0.05

 The highest mean score given by evaluators to residents belongs to the Coworkers group (Mean ±SD=149.6 ±17.23). Based on the General Linear Model (Repeated Measure), the differences between the mean scores given by the four groups of assessors were significant(P<0.05) (Table2).

 

Table 2: Trend of the scores obtained from the assessors

Assessors

Min

Max

Mean

SD

Self-assessment

123/8

166

143/7

±11/55

peer

101/7

171/7

134/4

±15

faculty member

93/5

157/4

135/1

±13

nurse

102/9

170

149/6

±17/23

 Although it was not statistically significant, the nurses and medical staffs gave a better score to the second -years than third –year residents (P>0.05).

The third –year residents` Self-assessment mean scores were significantly higher in comparison with the second –years by using T-test (P value=0.01), although no significant differences were observed among the scores which were given to both groups by coworkers, peers and faculty members (Table 3).

Table 3: Comparison of the obtained scores in the second and third year residents.

Assessor

 

Second year

Third year

Mean

SD

Mean

SD

Self-assessment

139.73

±9.3

147.7

±12.3

Peer

133.25

±13.9

135.6

±16.3

Faculty member

132.85

±13.5

137.26

±12.5

Nurse

151

±15/6

148/4

±18/9

 

DISCUSSION

The 360- degree evaluation method is a better assessment method because it includes perspective of assessor about full spectrum of the resident’s behavior.

In this study, through a modified 360° multisource feedback model, the second -year and third –year residents of general surgery, urology and orthopedics and neuro surgery competencies were evaluated. Not only this study was conducted for the first time in Shiraz, but also considering the acceptable Cronbach's α, it resulted in a reliable Persian translation of 360° multisource feedback questioners which provided us a perfect adapted instrument for our academic hospitals. The validity was obtained using the face validity which is similar to other studies and the same competencies (professionalism, communication skill and clinical performance and patient care were evaluated) (2, 3, 12,16). The results of this study suggest that the 360° multisource feedback model is a reliable, valid and feasible method for the residents’ evaluation; this result has confirmed the findings of other studies (3, 9,14). Considering the high voluntary participation and response rate, it seems that the residents found the implementation of this evaluation essential. This was consistent with other studies` findings (3, 9,15). Similar to other studies, our findings showed that using the 360° multisource feedback model is beneficial in the evaluation of the residents’ main competencies especially professionalism (9, 12, 13,16). In contrast with Joshi and Chandler reports, based on our findings, the residents who obtain higher grades and were clinically more skillful gave themselves higher scores) 11,17). This conflict is probably due to the training differences. Since Shiraz University of Medical Sciences’ hospitals are referral hospitals, each resident treat varied cases during their residency. They are also responsible for the supervision of many undergrads. These factors contributed to their higher self-confidence and in turn higher self-assessment. Our findings demonstrate that the third -year residents gave themselves higher scores in comparison with the second –years. This is probably due to the fact that they were more experienced and did more procedures. Therefore, they were more confident and used professionalism principles in their daily practices (21).

Unlike Massagli study, a significant relationship between the viewpoints of peers and faculty members were observed. This conflict might be because we used the same questionnaire for both groups since their aspect of evaluations were similar, as both groups had believed in the required clinical competences (12).

As other studies also stated, there was no significant relationship or very little correlation between other groups of the assessors (18).

Similar to other study, compared with the other evaluators, the Coworkers gave higher scores to the residents. (17). It shows that the Coworkers (nurses and staffs) mostly evaluate residents based on professionalism and communication skill (written or oral) and, while they can give good feedback regarding these general competences, more specific and clinical competences cannot be evaluated by them. These skills should be evaluated only by assessors with clinical insight such as faculties and peers.

This study was limited by the fact that it was performed on one filed (surgery), in a limited time period and the patients as an assessor was excluded. To enhance the psychometrics properties of the Persian translation of this questionnaire, re-conducting this study in other fields of specialty for a longer period of time is suggested.

Acknowledgement

The researcher appreciates all residents who provided us with their experiences.

 

Research committee approval and financial support:

This article is based on thesis done by Razieh Neshatavar as a  Partial Fulfillment of the Requirements for obtaining  a  Master Degree in  Medical Education and was approved in Shiraz University Medical Research Committee on April 9,2011  with approval number “ 90-5534 “

Conflict of interest

The author has no conflict of interest.

1- Hosseini Teshnizi S, Zare S, Solati M. Quality analysis of multiple choice questionnaire (MCQs) examination of non-continuous undergraduate medical records. Quarterly journal of Hormozgan University of Medical Sciences 2010; 14(3): 177-8. [In Persian].

2- Mery M, Greenberg JA, Patel A, Jaik NP. Teaching and assessing the ACGME competencies in surgical residency. Bull Am CollSurgeons 2008; 93(7): 39-48.

3- Violato C, Lockyer J, Fidler H. Multisource feedback: a method of assessing surgical practice. BMJ2003; 326: 546-8.

4- Rodgers KG, Manifold C. 360-degree feedback: Possibilities for assessment of the ACGME core competencies for emergency medicine residents. Accad Emerg Med 2002; 9(11): 1300-4.

5- Bidokhti Aliakbar A, Sharifi N. Developing a model based on 360 degree feedback for the evaluation of academic performance of students admitted without university entrance exam for graduate studies. Curriculum plannning knowledge and research in educational sciences 2014; 10(12): 118-33. [In Persian].

6- Berk RA1. Using the 360°s multisource feedback model to evaluate teaching and professionalism. Med Teach 2009; 31(12): 1073-80.

7- Raee H, Amini M, Momen Nasab A, Malek Pour A, Jafari MM. Team-based assessment of professional behavior in medical students. J Adv Med Educ Prof 2014; 2(3): 126-30.

8- Nofziger AC, Naumburg EH, Davis BJ, Mooney CJ, Epstein RM. Impact of peer assessment on the professional development of medical students: a qualitative study. Acad Med 2010; 85(1): 140-7.

9- Lockyer J, Violato C, Fidler H. A multi-source feedback program for anesthesiologists. Can J Anesth 2006; 53(1): 33-9.

10- Wilkinson TJ, Ed MC, Wade WB, Knock LD. A blueprint to assess professionalism: Results of a systematic review. Acad Med 2009; 84: 551-8.

11- Joshi R, Ling FW, Jaeger J. Assessment of a 360-degree instrument to evaluate residents’ competency in interpersonal and communication skills.  Acad Med 2004; 79(5): 458-63.

12- Massagli TL, Carline JD. Reliability of a 360-degree evaluation to assess resident competence. Am J Phys Med Rehabil2007; 86(10): 845-52.

13- Lelliott P, Williams R, Mears A, Andiappan M, Owen H, Reading P, et al. Questionnaires for 360-degree assessment of consultant psychiatrists: development and psychometric properties. Br J Psychiatry 2008; 193: 56-160.

14- Wood J, Collins J, Burnside E, Albanese MA, Propeck  PA, Kelcz F, et al. Patient, faculty, and self-assessment of radiology performance: A 360-degree method of measuring professionalism and interpersonal/communication skills. Acad Radiol 2004; 11: 931-9.

15- Higgins RS, Bridges J, Burke JM, O'Donnell MA, Cohen NM, Wilkes SB. Implementing the ACGME general competencies in a cardiothoracic surgery residency program using 360-degree feedback. Ann Thorac Surg  2004; 77(1): 12-7.

16- Violato C, Lockyer JM, Fidler H. Assessment of psychiatrists in practice through multisource feedback.  Can J Psychiatry 2008; 53(8): 525-33.

17- Chandler N, Henderson G, Park B, Byerley J, Brown WD, Steiner MJ. Use of a 360-degree evaluation in the outpatient setting: The usefulness of nurse, faculty, patient/family, and resident self-evaluation. J Grad Med Educ 2010; 2(3): 430-5.

18- Ogunyemi D, Gonzalez G, Fong A, Alexander C, Finke D, Donnon T, et al. From the eye of the nurses: 360-degree evaluation of residents. J Contin Educ Health Prof  2009; 29(2): 105-10.

19- Amini M, Abiri S, Nabeiei P,  Ghanavati SH, Hayat AA, Kojuri J. Assessment of the interns’ ability based on Dundee model in Shiraz University of Medical Sciences. J Adv Med Educ Prof 2015; 3(4): 189-95.

20- Emadzadeh A,Farzad F, Gholami H. Development of a framework for assessing professionalism in medical students. Future of medical education journal 2014; 4(1-2): 9-10.

21- Nemati Sh, Saberi A, Heydarzadeh A. Survey of the attitudes of residents of Guilan University of Medical Sciences toward Medical professionalism, and their professional behaviors. Future of medical education journal 2013; 3: 2.