The Objective Structured Clinical Examination (OSCE): Is it a reliable and valid method in evaluating the knowledge and clinical practice of midwifery students?

INTRODUCTION

Objective structured clinical examination (OSCE), defined as a well structured method of assessment to evaluate clinical competence, focuses on the outcomes through observable behaviors (1). OSCE was first described by Rolad Harden in Scotland in the late 1970s (2, 3). It is performed in a consecutive series of stations and consists of a series of 5-10 minutes clinical tasks(4) through which all students are evaluated in a number of skills such as: history taking, physical examination, communication skills and attitudes. The ability of the students to undertake practical procedures may be evaluated, too. Students were observed by examiners who stayed in each station throughout the session, scored the performance on a structured marking sheet and merely interacted with the students for providing instructions or asking about predetermined operations (5-8).

Although, OSCE has been mentioned as an expensive (9, 10), time consuming(11, 12)and, in certain cases, a stressful method of evaluation (8, 13, 14), it meets acceptable standards of the content, and is less potentially biased, especially examiner biased; in addition, it has high level of reliable and valid examination which of course, depends on some elements such as: scenario and variety of contents, checklist and scoring methods, number of stations and examiners per station and simulated patients (7, 9, 14, 15).

The use of OSCE is widespread as a valid academic measurement of medical licensing examination and nursing competence in North America, Canada, Australia, the United Kingdom and United states (7, 14, 16).

In Iran, according to available evidence, however, the reliability and validity of OSCE was shown among the nursing students by Moattari M, et al (17); there is no study in midwifery students. The main purpose of this study was to determine the reliability and validity OSCE method in evaluating knowledge and clinical practices of midwifery students in the courses of physiopathology, infectious diseases, internal diseases and surgery of Islamic Azad University Karaj Branch–Iran.

METHODS

This descriptive-analytic study was conducted on 39 midwifery students at the end of the third year of their education in Islamic Azad University Karaj Branch –Iran– through the academic year 2011-2012. Eligible students were those studying midwifery who had passed the first and the second levels of the physiopathology, infectious diseases, internal diseases and surgery courses; and were studying at the third level of the above mentioned courses. The protocol was approved by the Ethical Committee of Islamic Azad University Karaj Branch. After explaining the objective of the study and verification of the inclusion and exclusion criteria, each subject signed written informed consent and those who did not agree to sign the informed consent were excluded from the study.

The preparation and designing of scenarios, station contents, checklists, etc were done with an intensive 5-month effort of a team of teaching staff on internal medicine, gynecology and midwifery from Mashhad, Tehran and Alborz Universities of Medical Sciences and Islamic Azad University Karaj Branch. The face and content validity of OSCE was established by the broad consensus on these issues. The team agreed on the following five practical objectives: the ability of history taking, physical examination, diagnosing of disease, interpretation of lab data and communication skills. These objectives were designed into 10 stations and scoring was assigned from 0 to 9-12 at each station (maximum 12 in two stations, 10 in four stations and maximum 9 in the remaining stations), so the maximum scoring of all the stations was 100. The content of each station and checklist which were designed by a group of staff were reviewed later for evaluating their accuracy in a session performed with the presence of all of our specialists. Finally, it was decided to perform the first objective (history taking) at the station 1 & 2, the second (physical examination) at the station 3 & 4, the third (diagnosing of disease) at the station 5 & 6 & 9, the fourth (interpretation of lab data) at the station 7 & 8, and finally the fifth objective (communication skills) at the station 10. We considered four interactive simulated patient stations (station 1, 2, 9 and 10), two stations by using moulage (station 3 and 4) and four stations on laboratory investigation (station 5, 6, 7 and 8). Table 1 shows the content of each station.

To standardize the scenarios being used on simulated patient stations, four nursing students were chosen from Islamic Azad University Karaj Branch and trained by the staff to play their roles as patients before examination. It took 5 minutes for each subject or participant to pass each station and 30 seconds for going from one station to another; therefore, it nearly took 60 minutes to complete the OSCE. Asking the students to stay in a room was one of our strategies to prevent the students from sharing their information. A guideline for executing the exam was attached to the examinee's chair on each station. For determining the accuracy of the observer's judgment on the examination based on the checklist, the inter-rater reliability of the assessors' scoring was assessed by using two observers at each of the station. Out of the 39 midwifery students who participated in this OSCE, 20 were selected randomly to participate in another OSCE which was performed a fortnight later to evaluate the reliability of the OSCE. The assessors, scene and the scenarios in the second exam were similar to the first. We measured internal reliability using Cronbach's alpha.

Table 1: Content of each station in OSCE exam.

 (®: Bone Marrow Aspiration)

RESULTS

The mean OSCE score (standard deviation) obtained at each of the objectives was as follows: 68.80 (13.10) in the scope of history taking, 80.09 (8.60) in the scope of physical examination, 35.26 (18.73) in the scope of diagnosing of disease, 49.39 (21.45) in the scope of interpretation of lab data, and finally 69.23 (20.82) in the scope of communication skills. The average of total score which was obtained from all the stations was 85.29 (8.9).

Inter-rater reliability of the assessors' scoring assessed by using two observers at each of the station showed a good correlation between the scores of assessors in all the scopes (table 2).

The internal reliability of the OSCE's five scope of objectives was measured using Cronbach's alpha which is shown in table 3, and can be considered acceptable.

Table 2: The correlation between the scores of two assessors on the scope of objectives

Table 3: The internal reliability of the OSCE's five scope of objectives 

DISCUSION

The result of this study showed that the mean (standard deviation) of OSCE's total score obtained from all the stations was 85.29 (8.9). The highest score was obtained in the scope of communication skills and the lowest in the scope of diagnosis disease (data not shown). Regarding the reliability and according to the results measured by Cronbach's alpha, it can be said that the internal reliability of the OSCE is acceptable.

In a study on evaluating the knowledge and practice about drugs in the third-year nursing students, Dokoohaki R, et al. (18) have shown that the mean total score (standard deviation) of OSCE was 62.80 (7.71). In another study on senior nursing students in the cardiac intensive care unit, Mozafari M. (19) showed that the highest score belonged to the scope of history taking and the lowest to the scope of diagnostic skills and controlling the arrhythmia which was, to some extent, congruent with the results reported in our study.

Reliability which pertains to the reproducibility or consistency of a finding can be affected by many factors such as assessor's judgments, cases, nervousness of participants, and the whole conditions in which the test was performed (4). We repeated the OSCE on a fortnight later to evaluate the reliability. Our assessors, cases, the scene and scenarios were the same as the test was performed first. Since the nature of the individual competency and personality is unique and since a stressful manner can adversely affect the performance of OSCE, we randomly selected 50% of the participants to perform the second OSCE, in order to create a congruent situation and omit the effect of different competency and stress; however, we cannot claim that its impression could be well omitted in our study. Yet, some studies attest that validity of OSCE will increase in stressful situations which are usually seen in real world of clinical practice (7). Brosnan M, et al. believe that waiting in a corridor beforehand will be concomitant with the highest stress levels (14). We asked the students to stay in a room to prevent them from sharing their information; it, however, might have created a stressful situation before beginning the exam, but it can be accepted as it was presented in situations, the first and the second examination.   

Inter-rater reliability of OSCE which can be measured by a correlation of the assessors' scoring at each selected station, within a range from 0 which means no relationship between two phenomena to 1, signifying a perfect agreement. However, scoring 0.8 and above was considered as "gold standard", a good level of agreement was seen at the scores of 0.6 and above (7). In our study we expressed the highest correlation (0.90) in physical examination and the lowest (0.66) in communication skills; which can overall be considered acceptable. The result of this study is congruent with Jain et al. that showed in their study a correlation of (0.73-0.96) in history taking and (0.84-0.88) in neurological physical examination (20).

Internal consistency is another important test to evaluate the reliability of OSCE. Newble (6) mentioned that high level of reliability is achieved by using a long examination. On the other hand, using a variety of different stations seems to be more important in achieving maximum internal consistency (21). Meanwhile, the higher internal consistency would be achieved when the higher homogeneity of task is presented at different stations or the higher similarity is achieved on the skills tested (1). When the OSCE is supplemented by a writing-based station (16), the highest reliability would be achieved. Brannick MT, et al showed that the reliability to the station was about 0.56 when 10 stations or less were used and it increased to 0.74 when more than 10 stations were used (21). Yet, in some studies with less than 10 stations a reliability of > 0.80 is reported too; on the contrary, a reliability of < 0.80 was obtained in some studies with more than 25 stations (21). It is believed that more than 35 stations are needed to achieve an alpha of more than 0.8, which does not seem to be practical (22). In the present study the objectives were designed into 10 stations and a range between 0.52 and 0.78 was shown by Cronbach's alpha as a reliability index which can be considered acceptable. Our finding was congruent with the report of Brannick MT, et al. when 10 stations were used.

This study had some limitations. The most important limitation was the number of participants which was relatively low. The result of this study showed that OSCE can be considered as a reliable method of assessing clinical skills in midwifery students. Nevertheless, more research is needed to clarify the best method of OSCE in which the maximum reliability of the test would be achieved.

Acknowledgment

The authors would like to thank all the faculty members and midwifery students of Islamic Azad University Karaj Branch–Iran, who participated in this study. We are also grateful to the Vice Chancellor for Education and Research of Alborz University of Medical Sciences for funding and supports.

Disclosure statement: The authors have declared no conflicts of interest.