The Objective Structured Clinical Examination (OSCE): Is it a reliable and valid method in evaluating the knowledge and clinical practice of midwifery students?

Document Type: Original Article

Authors

1 Department of Midwifery, Karaj Branch, Islamic Azad University, Karaj, Iran

2 Department of Rheumatology, Mashhad University of Medical Sciences, Mashhad, Iran

3 Medical Education Research Center, Department of Statistics and Epidemiology, Faculty of Health, Tabriz University of Medical Sciences, Tabriz, Iran

4 Research Management, Mashhad University of Medical Sciences, Mashhad, Iran

5 Department of Pediatrics, Alborz University of Medical Sciences, Karaj, Iran.

Abstract

Background:
Objective structured clinical examination (OSCE) is one of the best methods for evaluating the achievement of educational objectives and cognitive, emotional and psycho-motor aspects of medical students. The main purpose of this study was to determine the reliability and validity of OSCE method in evaluating knowledge and clinical practices of midwifery students in the courses of physiopathology, infectious diseases, internal diseases and surgery of Islamic Azad University Karaj Branch –IRAN.
Methods:
This descriptive-analytic study was conducted on 39 midwifery students at the end of the third year of their education in Islamic Azad University Karaj Branch –IRAN through the academic year 2011-2012. Students performed 10 different OSCE stations based on five practical purposes designed by medical experts on curriculum in this field. For the face and content validity of OSCE, the broad consensus on these issues was used. In order to determine the accuracy of the observer's judgment, the inter-rater reliability and for internal reliability test-retest and Cronbach's alpha were used.
Results:
The mean of total OSCE score (standard deviation) was 85.29 (8.9). The face and content validity was confirmed after numerous meetings and discussion with teaching staffs on internal medicine and extracting practical purposes. Moreover, the reliability was confirmed by parallel validity, test-retest and Cronbach's alpha. The Cronbach's alpha in five scope of OSCE showed a range between 0.52 and 0.78 which is considered acceptable.
Conclusion: Implementation of OSCE is suggested as a reliable and valid means of evaluating knowledge and clinical practice of midwifery students.

Keywords


INTRODUCTION

Objective structured clinical examination (OSCE), defined as a well structured method of assessment to evaluate clinical competence, focuses on the outcomes through observable behaviors (1). OSCE was first described by Rolad Harden in Scotland in the late 1970s (2, 3). It is performed in a consecutive series of stations and consists of a series of 5-10 minutes clinical tasks(4) through which all students are evaluated in a number of skills such as: history taking, physical examination, communication skills and attitudes. The ability of the students to undertake practical procedures may be evaluated, too. Students were observed by examiners who stayed in each station throughout the session, scored the performance on a structured marking sheet and merely interacted with the students for providing instructions or asking about predetermined operations (5-8).

Although, OSCE has been mentioned as an expensive (9, 10), time consuming(11, 12)and, in certain cases, a stressful method of evaluation (8, 13, 14), it meets acceptable standards of the content, and is less potentially biased, especially examiner biased; in addition, it has high level of reliable and valid examination which of course, depends on some elements such as: scenario and variety of contents, checklist and scoring methods, number of stations and examiners per station and simulated patients (7, 9, 14, 15).

The use of OSCE is widespread as a valid academic measurement of medical licensing examination and nursing competence in North America, Canada, Australia, the United Kingdom and United states (7, 14, 16).

In Iran, according to available evidence, however, the reliability and validity of OSCE was shown among the nursing students by Moattari M, et al (17); there is no study in midwifery students. The main purpose of this study was to determine the reliability and validity OSCE method in evaluating knowledge and clinical practices of midwifery students in the courses of physiopathology, infectious diseases, internal diseases and surgery of Islamic Azad University Karaj Branch–Iran.

METHODS

This descriptive-analytic study was conducted on 39 midwifery students at the end of the third year of their education in Islamic Azad University Karaj Branch –Iran– through the academic year 2011-2012. Eligible students were those studying midwifery who had passed the first and the second levels of the physiopathology, infectious diseases, internal diseases and surgery courses; and were studying at the third level of the above mentioned courses. The protocol was approved by the Ethical Committee of Islamic Azad University Karaj Branch. After explaining the objective of the study and verification of the inclusion and exclusion criteria, each subject signed written informed consent and those who did not agree to sign the informed consent were excluded from the study.

The preparation and designing of scenarios, station contents, checklists, etc were done with an intensive 5-month effort of a team of teaching staff on internal medicine, gynecology and midwifery from Mashhad, Tehran and Alborz Universities of Medical Sciences and Islamic Azad University Karaj Branch. The face and content validity of OSCE was established by the broad consensus on these issues. The team agreed on the following five practical objectives: the ability of history taking, physical examination, diagnosing of disease, interpretation of lab data and communication skills. These objectives were designed into 10 stations and scoring was assigned from 0 to 9-12 at each station (maximum 12 in two stations, 10 in four stations and maximum 9 in the remaining stations), so the maximum scoring of all the stations was 100. The content of each station and checklist which were designed by a group of staff were reviewed later for evaluating their accuracy in a session performed with the presence of all of our specialists. Finally, it was decided to perform the first objective (history taking) at the station 1 & 2, the second (physical examination) at the station 3 & 4, the third (diagnosing of disease) at the station 5 & 6 & 9, the fourth (interpretation of lab data) at the station 7 & 8, and finally the fifth objective (communication skills) at the station 10. We considered four interactive simulated patient stations (station 1, 2, 9 and 10), two stations by using moulage (station 3 and 4) and four stations on laboratory investigation (station 5, 6, 7 and 8). Table 1 shows the content of each station.

To standardize the scenarios being used on simulated patient stations, four nursing students were chosen from Islamic Azad University Karaj Branch and trained by the staff to play their roles as patients before examination. It took 5 minutes for each subject or participant to pass each station and 30 seconds for going from one station to another; therefore, it nearly took 60 minutes to complete the OSCE. Asking the students to stay in a room was one of our strategies to prevent the students from sharing their information. A guideline for executing the exam was attached to the examinee's chair on each station. For determining the accuracy of the observer's judgment on the examination based on the checklist, the inter-rater reliability of the assessors' scoring was assessed by using two observers at each of the station. Out of the 39 midwifery students who participated in this OSCE, 20 were selected randomly to participate in another OSCE which was performed a fortnight later to evaluate the reliability of the OSCE. The assessors, scene and the scenarios in the second exam were similar to the first. We measured internal reliability using Cronbach's alpha.

 

Number of the station

Content of the station

1

(simulated patient)

Maryam Talebi is a 60-year-old woman who visits the emergency ward because of chest pain and nausea. In the next 5 minutes take focused history and address her concerns.

2

(simulated patient)

Mahin Javidi, a 22-year-old and 24-week pregnant woman has been referred to clinic for checking plasma (blood) glucose. In the next 5 minutes take focused history.

3

(moulage)

A 21-year-old man visits clinic complaining of shortness of breath. His father has allergic rhinitis. In the next 5 minutes perform focused physical examination.

4

(moulage)

A 45-year-old woman visits clinic because of sensation of a mass on her right breast. Perform focused physical examination in the next 5 minutes.

5

(photo and X-Ray)

A 55-year-old woman visits clinic due to right wrist swelling and pain. Her hand’s photo (Power point slide) and her PA and lateral right hand x-ray on negatoscope are shown. Explain her photo and x-rays and address her concerns in the next 5 minutes.

6

(X-Ray and Lab data )

A 66-year-old man visits clinic complaining of bone pain, weight loss and nausea. His scalp x-ray is shown on negatoscope. His lab exam shows:

Hb= 8 mg/dl     MCV= 89 fL     MCH= 32 pg        PLT: 380000/µL        ESR= 80 mm

BMA®= 20% plasma cell       Serum Iron=40  µg/dl      Serum Ferritin=50  ng/dl   TIBC=30 µg/dl 

Explain his x-ray, abnormal lab data and address his concerns in the next 5 minutes.

7

(interpretation of lab data)

Photo of a 27-year-old woman who has 3 recurrent abortions and lower leg venous thrombosis, in her past history, is shown. Her lab exam shows:

Hb= 12 mg/dl            MCV & MCH = normal              ESR= 60 mm

ANA=positive                  Anti ds-DNA=140             

Anti phospholipid Ab (IgG)=positive

Explain her photo and lab data and address her concerns in the next 5 minutes.

8

(interpretation of lab data)

A 62-year-old man visits emergency ward complaining of acute chest pain with nausea, vomiting and sweating. His electrocardiogram (Power point slide) is shown. Explain abnormal findings in his electrocardiogram and address his concerns in the next 5 minutes.

9

(simulated patient)

Mina Taheri is a 37-year-old woman who visits clinic due to tingling and numbness of both hands, specially the right hand, which wakes her up during night. In the next 5 minutes perform focused physical examination and address her concerns.

10

(simulated patient)

Sara Nazemi is a 32-year-old woman visits clinic for consultation. She had gestational diabetes mellitus on her last pregnancy. She had her delivery 2 months ago. Her mother is a known case of type 2 diabetes mellitus. In the next 5 minutes, talk to her and give your advice on controlling her diabetes.

 

Table 1: Content of each station in OSCE exam.

 (®: Bone Marrow Aspiration)

  

RESULTS

The mean OSCE score (standard deviation) obtained at each of the objectives was as follows: 68.80 (13.10) in the scope of history taking, 80.09 (8.60) in the scope of physical examination, 35.26 (18.73) in the scope of diagnosing of disease, 49.39 (21.45) in the scope of interpretation of lab data, and finally 69.23 (20.82) in the scope of communication skills. The average of total score which was obtained from all the stations was 85.29 (8.9).

Inter-rater reliability of the assessors' scoring assessed by using two observers at each of the station showed a good correlation between the scores of assessors in all the scopes (table 2).

The internal reliability of the OSCE's five scope of objectives was measured using Cronbach's alpha which is shown in table 3, and can be considered acceptable.

 

OSCE's Scope of Objectives

Correlation Between the Scores of two Assessors

P-value

history taking

0.751

≤ 0.001

physical examination

0.901

≤ 0.001

diagnosing of disease

0.773

≤ 0.001

interpretation of lab data

0.715

0.01

communication skills

0.660

0.01

Table 2: The correlation between the scores of two assessors on the scope of objectives

 

OSCE's Scope of Objectives

Cronbach's alpha

history taking

0.627

physical examination

0.623

diagnosing of disease

0.784

interpretation of lab data

0.649

communication skills

0.525

 

Table 3: The internal reliability of the OSCE's five scope of objectives 

DISCUSION

The result of this study showed that the mean (standard deviation) of OSCE's total score obtained from all the stations was 85.29 (8.9). The highest score was obtained in the scope of communication skills and the lowest in the scope of diagnosis disease (data not shown). Regarding the reliability and according to the results measured by Cronbach's alpha, it can be said that the internal reliability of the OSCE is acceptable.

In a study on evaluating the knowledge and practice about drugs in the third-year nursing students, Dokoohaki R, et al. (18) have shown that the mean total score (standard deviation) of OSCE was 62.80 (7.71). In another study on senior nursing students in the cardiac intensive care unit, Mozafari M. (19) showed that the highest score belonged to the scope of history taking and the lowest to the scope of diagnostic skills and controlling the arrhythmia which was, to some extent, congruent with the results reported in our study.

Reliability which pertains to the reproducibility or consistency of a finding can be affected by many factors such as assessor's judgments, cases, nervousness of participants, and the whole conditions in which the test was performed (4). We repeated the OSCE on a fortnight later to evaluate the reliability. Our assessors, cases, the scene and scenarios were the same as the test was performed first. Since the nature of the individual competency and personality is unique and since a stressful manner can adversely affect the performance of OSCE, we randomly selected 50% of the participants to perform the second OSCE, in order to create a congruent situation and omit the effect of different competency and stress; however, we cannot claim that its impression could be well omitted in our study. Yet, some studies attest that validity of OSCE will increase in stressful situations which are usually seen in real world of clinical practice (7). Brosnan M, et al. believe that waiting in a corridor beforehand will be concomitant with the highest stress levels (14). We asked the students to stay in a room to prevent them from sharing their information; it, however, might have created a stressful situation before beginning the exam, but it can be accepted as it was presented in situations, the first and the second examination.   

Inter-rater reliability of OSCE which can be measured by a correlation of the assessors' scoring at each selected station, within a range from 0 which means no relationship between two phenomena to 1, signifying a perfect agreement. However, scoring 0.8 and above was considered as "gold standard", a good level of agreement was seen at the scores of 0.6 and above (7). In our study we expressed the highest correlation (0.90) in physical examination and the lowest (0.66) in communication skills; which can overall be considered acceptable. The result of this study is congruent with Jain et al. that showed in their study a correlation of (0.73-0.96) in history taking and (0.84-0.88) in neurological physical examination (20).

Internal consistency is another important test to evaluate the reliability of OSCE. Newble (6) mentioned that high level of reliability is achieved by using a long examination. On the other hand, using a variety of different stations seems to be more important in achieving maximum internal consistency (21). Meanwhile, the higher internal consistency would be achieved when the higher homogeneity of task is presented at different stations or the higher similarity is achieved on the skills tested (1). When the OSCE is supplemented by a writing-based station (16), the highest reliability would be achieved. Brannick MT, et al showed that the reliability to the station was about 0.56 when 10 stations or less were used and it increased to 0.74 when more than 10 stations were used (21). Yet, in some studies with less than 10 stations a reliability of > 0.80 is reported too; on the contrary, a reliability of < 0.80 was obtained in some studies with more than 25 stations (21). It is believed that more than 35 stations are needed to achieve an alpha of more than 0.8, which does not seem to be practical (22). In the present study the objectives were designed into 10 stations and a range between 0.52 and 0.78 was shown by Cronbach's alpha as a reliability index which can be considered acceptable. Our finding was congruent with the report of Brannick MT, et al. when 10 stations were used.

This study had some limitations. The most important limitation was the number of participants which was relatively low. The result of this study showed that OSCE can be considered as a reliable method of assessing clinical skills in midwifery students. Nevertheless, more research is needed to clarify the best method of OSCE in which the maximum reliability of the test would be achieved.

Acknowledgment

The authors would like to thank all the faculty members and midwifery students of Islamic Azad University Karaj Branch–Iran, who participated in this study. We are also grateful to the Vice Chancellor for Education and Research of Alborz University of Medical Sciences for funding and supports.

Disclosure statement: The authors have declared no conflicts of interest.

1)      Carraccio C, Englander R. The objective structured clinical examination: A step in the direction of competency based evaluation. Arch Pediatr Adolesc Med 2000; 154: 736-41.

2)      Harden RM, Stevenson M, Downie WW, Wilson GM. Assessment of clinical competence in using objective structured examination. Br Med J 1975; 1: 447-51.

3)      Harden RM, Gleeson FA. Assessment of clinical competence using an objective structured clinical examination (OSCE). Med Educ 1979; 13 (1): 41-54.

4)      Wass V, Van der Vleuten C, Shatzer J, Jones R. Assessment of clinical competence.Lancet 2001; 357: 945-49.

5)      Townsend A, McLlveny S, Miller C, Dunn E. The use of an objective structured clinical examination (OSCE) for formative and summative assessment in general practice clinical attachment and its relationship to final medical school examination performance. Med Educ 2001; 35: 84-6.

6)      Newble D. Techniques for measuring clinical competence: objective structured clinical examination. Med Educ 2004; 38(2): 199-203.

7)      Rushforth H. Objective structured clinical examination (OSCE): review of literature and implications for nursing education. Nurs Educ Today 2007; 27: 481-90.

8)      Selim AA, Ramadan FH, El-Gueneidy MM, Gaafer MM. Using Objective Structured Clinical Examination (OSCE) in undergraduate psychiatric nursing education: Is it reliable and valid? Nurs Educ Today 2012; 32: 283-8.

9)      Turner J, Dankoski M. Objective structured clinical exams: a critical review. Fam Med 2008; 40(8): 574-8.

10)  Wallace J, Rao R, Haslam R. Simulated patients and objective structured clinical examinations: a review of their use. Adv Psychiatr Treat 2002; 8: 342-8.

11)  Major D. OSCEs-seven years on the bandwagon: the progress of an objective structured clinical evaluation programme. Nurs Educ Today 2005; 25: 442-54.

12)  Byrne E, Smyth S. Lecturers' experiences and perspectives of using an objective structured clinical examination. Nurs Educ Pract 2008; 8: 283-9.

13)  Nicol M, Freeth D. Assessment of clinical skills: a new approach to an old problem. Nurs Educ Today 1998; 18: 601-9.

14)  Brosnan M, Evans W, Brosnan E, Brown G. Implementing objective structured clinical skills evaluation (OSCE) in nurse registration programs in a centre in Ireland: A utilization focused evaluation. Nurs Educ Today 2006; 26: 115-22.

15)  Wass V, Roberts C, Hoogenboom R, Jones R, Van der Vleuten C. Effect of ethnicity on performance in a final objective structured clinical examination: qualitative and quantitative study. BMJ 2003; 326: 800-3.

16)  Griesser MJ, Beran MC, Flanigan DC, Quackenbush M, Van Hoff C,  Bishop JY. Implementation of an Objective Structured Clinical Exam (OSCE) into orthopedic surgery residency training. J Surg Educ 2012; 69: 180-9.

17)  Moattari M, Abdollah-zargar S, Mousavinasab M, Zare N, Beygi Marvdast P. Reliability and validity of OSCE in evaluating clinical skills of nursing students. J Med Educ2009; 13(3): 79-85.

18)  Dokoohaki R, Sharafi N, Rahnema Sh, Azar Hooshang P, Jahanbin I. Evaluation of the knowledge and practice of the third year nursing student about drugs by OSCE method. Iranian J Nurs2008; 21(55): 101-109. [Persian].

19)  Mozafari M. Practice of senior nursing student in the cardiac intensive care with using OSCI exam. Journal of Ilam University of Medical Science 2003; 12: 45-52. [Persian].

20)  Jain S, DeLisa J, Eyles M, Nadler S, Kirshblum S, Smith A. Further experience in development of an objective structured clinical examination for physical medicine and rehabilitation residents. Am J Phys Med Rehabil1998; 77(4): 306-10.

21)  Brannick MT, Erol-Korkmaz HT, Prewett M. A systematic review of the reliability of objective structured clinical examination scores. Med Educ 2011; 45: 1181-9.

22)  Martin IG, Jolly B. Predictive validity and estimated cut score of an objective structured clinical examination (OSCE) used as an assessment of clinical skills at the end of the first clinical year. Med Educ2002; 36: 418-25.