The Goal-Oriented Evaluation of Medical Students during Internship Based on Performance Rating Tables

Introduction

Evaluation is a systematic and instrumental examination in order to enhance the efficiency of educational programs (1, 2). Different methods and approaches have been proposed to evaluate the students’ skills and performance during their internship and practicum which is the beginning of their entrance to medical and health service centers (3-5). Internship and practicum courses play an important role in enhancing the students’ skills, also educational rotation of medical student in health service centers has key role in learning special subject like social accountability, managing skill and prevention strategies (1, 6). In clinical students’ training groups that most education and practices take place without direct daily supervision of the instructors and the students’ performance is highly dependent on their personal interest and enthusiasm, smart written curriculum is needed to manage the education process, monitor the students’ performance, and evaluate their progress (2, 3). According to the conducted studies, students have limited knowledge about the educational objectives, expected performances, the orders of the plans, and what is considered in evaluation. On the other hand, since most conventional methods evaluate superficial knowledge, ignore positive and negative feedbacks; they are not suitable measures for evaluating clinical students’ performance (2, 7).

According to conventional definitions, performance is the result of comparing the present situation with expectations, reaching achievements and comparing them with objectives; therefore, evaluating the students’ performance is a process that deals with assessing and judging the students’ behavior and skills during a specific period (1, 8). Subjective judgments like personal judgment that has no specific criteria and is the same for all similar situations are not affirmed by others and have no quantitative reliability. Conversely, objective judgment which is utilized for visible factors like quantity or quality of work, behaviors, and responsibilities has acceptable reliability (4, 9).

In evaluating the students, unstructured methods like assignments and essay responses may require more skill and creativity from both the students and the evaluator whilst structured methods possess higher reliability and validity in assigning scores (10). However, curriculum-based assessment, performance assessment, and ecological assessment combine structured and unstructured informal assessments (11). Checklist and performance rate scale are among useful tools that utilize observation method to systematically objectify the evaluation and assessment of the students’ scientific skills and attitudes (11, 12). Checklists and performance rating table are prefabricated lists of questions and instructions that should be followed in order to achieve a particular objective or any skill or activity which can be divided and classified into a number of smaller skills and activities or specific observable behaviors whose occurrence or absence of those is easily determinable (12-14). Basic components and subjects are involved in the checklist, for scoring the items of each component, accomplished or not accomplished, and behavior quality (usually with 4-8 Likert rating) are employed. Based on the questions or items and expected conditions, the evaluator should score behavioral objectives and must-to-learn materials in a graded table (12, 13). Using the checklist and graded tables reduces the effects of the evaluator’s judgment on assessing the students’ skill and helps the instructor to discover the student’s strengths and weaknesses (8, 14). Developing of performance rating tables is inexpensive and applicable in a short time and can provide the immediate feedback (12, 13).

The present study assessed the effectiveness of evaluating the medical students during internship based on performance rating tables (PRTs).

Methods

This study was conducted in Community Medicine department of Mashhad University of Medical Sciences in 2012. In Community Medicine department, internship course for medical students lasts one month. In the first week of this period, the students are trained in the wards. In the following three weeks, they have a health center rotation as their internship task. After the internship curriculum was figured out in Community Medicine department, the objectives and the students’ responsibilities and suitable abilities were determined. The flowchart, checklist and PRTs were designed according to the curriculum and educational objectives. Two tables were designed for every week, and a total of 6 tables were designed for the whole course.

In developing the PRT, important educational features were included. The evaluated features were selected in a way that they were directly visible. The criteria, information, evidence, or activities for which the students were given scores were clearly defined. The range of the scores was determined, and an option was considered for cases that the evaluator could not judge on. The initial tables were given to the department professors and ambiguous cases were revised. The revised cases included determining the supervisor’s responsibilities, community assessment, determining the problems of community and their priorities, measuring the health index of population, managing the outpatients and assessing the population risk factors, and administering an educational plan for one of the target groups of population.

After the confirmation of the study, the supervisors of the health centers who participate in supervised of interns’ activities in the healthcare centers received a letter about details of program from the chairman of the Community Medicine department. Moreover, the health centers supervisors’ views were also applied in conducting the program. A copy of the flowchart and PRTs was provided for each center. Furthermore, in each course, the students received the chart of their tasks and their evaluation points in order to make them familiar with the program.

 At the beginning of each course, the students were explained about the process and their responsibilities. In order to make the observations concrete, the evidence based on which the students were expected to act and observations of their activities was prepared in their portfolio. The professors scored the students’ performance in each field recorded in portfolio based on their performance, interviews, responses, observation, and comparison with the expected performance in each field. The results of the scoring process were recorded in tables that were prepared for each group. During observation, the mistakes and problems of the students’ performance were noted and the results were provided for the supervising instructor in order to resolve.

The students’ mean score of their 3-week presence at the healthcare center was calculated. The results of the PRTs and the portfolio included 12 points out of 20; also 8 points were allotted to an essay exam. The effectiveness of the PRTs were assigned based on the students’ performance scores and also the students’ views about its effectiveness. At the end of the internship period, the students filled out a questionnaire containing questions on the effectiveness of the PRTs regarding to familiarize them with their responsibility, and how much changes occurred in their knowledge level, attitude, skills and abilities in the Community Medicine course. The questionnaire rating was 0-20. Moreover, the health centers supervisors’ evaluated the students’ level of activity, active participation, interest, and creativity in the Likert scale. The collected data were analyzed using SPSS 16.0. Mann-Whitney, Spearman correlation and Chi-square tests were applied for analysis and significance statistical level was determined lower than 0.05.

Results

The participants of the study were 173 medical students studying in their 6^th and 7^th year of their course and spending their internship in the Community Medicine department. Sixty-seven students were male (38.7%) and 106 were female (61.3%). Examining the quantitative variables through Shapiro-Wilk test indicates that their distribution was not normal.

According to the PRTs, the students’ performance mean score in health center rotation was 6.7±0.9 out of 8. Compared to the male students, the female students’ score was 0.4 points higher. Mann-Whitney test indicated that the difference between the scores in terms of gender was significant (P=0.01).

Assessment of the students’ views about PRTs effectiveness showed that the mean score of the performance rating tables’ effectiveness in familiarizing the students with their responsibilities was 17.2±1.4 from 20 point, also students scored to changing their knowledge, attitude and performance toward educational objective of Community Medicine course 18.1±1.1, 17.9±1.5, and 17.4±0.9 respectively. In comparison with the male students, the female students reported higher scores in all of above field, the results of Mann-Whitney indicated that there was a significant difference between the scores in regard with the participants’ gender (P<0.001). Table 1 presents the students’ scores on the effect of the PRTs and changing their knowledge, attitude and performance. Spearman test indicated a significant correlation between the effectiveness score of the PRTs and the final interns’ performance scores in the rotation (P<0.001, r=0.52), also with self-evaluation of their knowledge change (P<0.001, r=0.42), attitude change (P<0.001, r=0.41), and performance change (P<0.001, r=0.52). The interns’ knowledge, attitude, and performance had a significant relationship with each other, which is indicated in Table 1.

Spearman test proved a significant relationship between the effectiveness scores of the PRTs and the students’ activity scores in the health centers (P<0.001, r=0.42), participation (P=0.02, r=0.23), and interest and creativity (P<0.001, r=0.31) in supervisors’ view.

The health centers supervisors’ views on the interns’ activity level, participation, interest, and creativity based on Likert scale are presented in Table 2.

Based on the supervisors’ evaluations, Mann-Whitney test indicated no significant difference between the interns’ activity level, participation, and interest in terms of their gender (P>0.05).

Table 1. The correlation between the interns’ knowledge, attitude, and performance

Table 2. The health centers supervisors’ views on the medicine interns’ activity level, participation, and interest level by application of PRTs

Discussion

The results of the present study indicated that the "performance rating tables" are appropriate tools to evaluate social medicine interns in healthcare centers and can be utilized to manage educational programs in the field and evaluate the interns in healthcare centers.

The checklist reduces the distribution of the judgments and increases the consistency of the pursuits with the educational objectives and regulations. It also helps the evaluators to take all aspects into account (11, 12). Advantages of developing checklists and utilizing PRTs in management of a program include simple preparation and real evaluation of the performance and behaviors. If components are defined clearly and can be seen easily, the checklist can be reliable (11, 12).

In the present study, the students’ evaluation of the effectiveness of the evaluation tables was examined. They reported a mean score of 17.2±1.4 (out of 20) for the effectiveness of the performance rating tables. The results of a study conducted in order to compare the students’ performance scores and their satisfaction through two methods of checklists and logbook among midwifery students of Guilan faculty in 2005 indicated that there was a significant difference between the two methods; with 16.33±0.77 and 17.88±0.41 for functional skills logbook and checklists, respectively. Although all of the students were fully satisfied with the procedures of assessment in the checklist method, their satisfaction dropped to 85.7% after their scored were announced to be low (8).

In the present study female interns gained higher scores than male students, female also gave higher scores to the PRTs and had given higher scores to their abilities too, which indicates that female interns had showed higher interest to the proposed step-by-step program and had clearer views on their evaluation.

There was an appropriate correlation between the effectiveness of the PRTs and the interns’ gained scores in each single activity. This finding shows appropriate conduction of the activities in the field by the students who had familiarized with the PRTs well and were satisfied with them. There was correlation between the students’ performance and the effectiveness of the PRTs; therefore, it can be concluded that more familiarity with PRTs can cause the students to get more involved with the activities and participate in the program more actively. According to these findings, it can be stated that assessing the interns based on PRTs can enhance the quality of their performance in the field of health centers and the same educational filed.

Moreover, PRTs result in homogenizing the evaluation process and objectify of the scoring procedure of the students. Objective-oriented supervision of the interns’ activities and observation-based feedback can also be provided through this tools and method.

However, completing the PRTs requires trained evaluators who are skilled in precise and uniform evaluation (11, 12). Content validity of the PRTs depends on the importance of the questions. The validity of the instrument can be boosted through the scholars’ agreement over the main behavior and performances, the order of the activities, and evaluation criteria (11-13). Reliability of the scores gained from the PRT is another major problem, which should be dealt with through the agreement among the expert evaluators (11, 14). Clarifying the details of the components and items is essential in reliability enhancement and exact explanation of a performance based on the collected evidence from the interns, which was utilized in the present study.

Acknowledgments

Authors would like to acknowledge Medical Interns whose participation made this study possible.

Research committee approval and financial support: Authors would like to thank Mashhad University of Medical Sciences for the financial support for this study.

Conflict of interest: None to be declared.