Assessing the clinical skills of interns and stagers in the department of obstetrics and gynecology using Direct Observation of Procedural Skills (DOPS) method and satisfaction level of learners and examiners

Document Type: Original Article

Authors

1 Department of Urology, Shahid Beheshti University of Medical Sciences, Tehran, Iran

2 Medical Student, Tehran University of Medical Sciences, Tehran, Iran

3 Department of Community Medicine, School of Medicine, Birjand University of Medical Sciences, Birjand, Iran

4 Education Development Center, Birjand University of Medical Sciences, Birjand, Iran

5 Department of Curriculum Development, Faculty of Psychology and Educational Sciences, Allameh Tabatabaei University, Tehran, Iran

6 Department of Gynecology and Obstetrics, Birjand University of Medical Sciences, Birjand, Iran

Abstract

Background and Objectives: Evaluation of clinical skills of intern and stager students is one of the most important tasks of clinical faculty members. One of the methods of assessment is direct observation of procedural skills. Therefore, the aim of this study was to evaluate the clinical skills of interns and stagers in department of obstetrics and gynecology using the Direct Observation of Procedural Skills (DOPS) method.
Methods: This is a cross-sectional descriptive-analytical study. 60 interns and stagers of the department of obstetrics and gynecology were selected by census method. According to the opinions of faculty members of medical universities, among the main procedures of women's departments, 11 procedures were selected for testing materials and a checklist for their evaluation was prepared. The student was observed by the examiner during the procedure in the actual work environment and the results were recorded according to a structured checklist and presented in an objective setting environment. The content validity of the checklist was approved by faculty members of obstetrics and gynecology. Reliability was calculated as 85% based on Cronbach's alpha. Data were analyzed using SPSS version 18 and Chi-square and independent t-test were used at a significance level of 0.05.
 
Results: A total of 60 students in the department of obstetrics and gynecology were evaluated by this method. The evaluation indicators were as expected in most cases at all stations. The highest expected and above expected cases were related to communication skills and requesting assistance if needed; while most below expected cases were related to knowledge of the indications, anatomy, and procedure’s techniques. The most disadvantage mentioned by the students was the technical ability and the most advantage was requesting help if needed. There was a significant difference between the mean score of satisfaction in learners (5.3 ± 2.92) and the mean score of satisfaction in the examiners (8.85 ± 0.93) (p> 0.05).
Conclusion: Due to the higher than expected value of students' clinical skills, the use of this method is not only an incentive for learning but also because the method and content of the test are directly related to clinical performance, it can direct efforts to learning.

Keywords

Main Subjects


Introduction

Today, emphasis on clinical and communication skills training and creating a proper attitude in medical students has been widely considered (1). Possessing skills in conducting clinical examinations are the essence of the medical profession. Acquiring the right skills in performing clinical procedures requires time, patience and practice in a suitable environment. Clinical skills are slowly gained and are depleted by the lack of practice (2, 3). One of the most important duties of clinical faculty members is the assessment of students' clinical skills. The development of clinical education methods makes the use of new and appropriate clinical evaluation methods a necessity. The use of new evaluation approaches that reinforce learning at the same time, are being more widely considered (4). There are two approaches to university education: deep and superficial. If depth-based assessment is designed, it can lead to in-depth learning. The superficial approach is based on minimum effort and acquiring the passing score and the desire to learn the habitual selected content, while in an in-depth approach, student tries to understand the materials, integrate the components into the whole, and to learn with one's own knowledge and apply it in real situations. (5) Although the majority of medical students' evaluation is clinical evaluation, but finding the appropriate method which is satisfactory to the students, has always been challenging. (6) Student evaluation in a clinical setting not only requires the proper functioning of psycho-motor skills, but also reflects the science, knowledge, judgment, and the ability of the person to react to changes (7). Therefore, to judge the student's achievement to a level of competence in practical skill, one must show the ability to perform essential clinical skills (8, 9). However, clinical assessment is measured at the level of performance. In the categorization of the Miller’s pyramid, one of the various methods for evaluating performance is direct observation of procedural skills (10). This method involves observing an intern during an action procedure that takes place on a real patient in an environmentally responsible manner. (11) Direct observation of the practice of medical interns in real patients is critical to assessing performance-based clinical skills. (12) DOPS is a new assessment tool at the workplace of the student, and studies have shown that it is a useful tool for evaluating practical skills (13) and has a positive impact on the learning approach (14).

The results of the study by Sharemi in 2011 revealed that using the clinical skills training center and the objective methods of student assessment skills, especially before entering the real clinical setting, is necessary for observing patients' rights and meeting the needs of students (15). Considering that one of the important issues of health and development in societies is the need for women to have reproductive health services and health care, it is necessary for physicians to have adequate experience for initial treatment. Some of these problems are related to pregnancy (16). The use of virtual realities in the training of obstetrics and gynecology and midwifery skills was confirmed in a 2002 study by Letteri, quoted by Matari 2007 (17). Due to the fact that in many rural and remote areas, access to specialists is not readily possible, there is the likelihood that general practitioners confront maternal delivery and unforeseen situations during professional activities. Therefore, education of all medical students in this field, which does not contradict the principle of adaptation of educational environments with religious standards, is necessary (18). Considering the fact that in Birjand University of Medical Sciences, the evaluation of the end of the course for students in internship and training courses has long been based on the same questions as the theoretical questions that were asked at the end of the students' section and so the students were not evaluated in terms of practical learning. Also, since a study has not been done in Birjand University of Medical Sciences so far, we have decided to conduct a research aimed at assessing the clinical skills of interns and female students using the direct observation of procedural skills (DOPS) method.

Methods: This is a cross-sectional descriptive analytical study. The study population consisted of 60 interns and stagers of obstetrics and gynecology department who were surveyed by census method. According to the opinions of faculty members of medical universities, among the main procedures of obstetrics and gynecology departments, 11 procedures were selected for testing materials and a two-dimensional evaluation checklist was prepared to collect information about the clinical skills assessment of students in the department of obstetrics and gynecology. The first part of the checklist contains evaluation indicators that include 11 options. These options include: 1- Knowledge of the indications, anatomy and technique of the procedure 2- Patient satisfaction 3. Preparation stage before the procedure 4. Anesthetizing and proper sedation 5. Observance of the conditions sterility 6. Technical capability (e. g. normal delivery) 7. Requesting help if necessary 8. Follow-up procedures 9. Communication skills 10. Professional behavior and attention to the patient 11. The general skill of the procedure. For each of the options, the score is assigned from zero to 10, the score zero is assigned to the unacceptable item, between 1 to 3: lower than expected, 4 to 6: moderate, 7 to 9 and 10 are considered as higher than expected. The second part of the checklist is related to the degree of students and examiner’s satisfaction which consisted of two questions.

In order to assess the students' clinical skills, innovative stations for evaluating interns and stagers in Birjand University of Medical Sciences were used, which included 10 clinical stations, the student was required to attend all stations and assessed by the physician at the station. During the questionnaire, the student's current problems are presented simultaneously. The ten stations include a history station, a gynecologic station, a Pap smear station, and a Pap smear report, a fetal heart hearing station and Leopold’s maneuvers, a urinary catheter station, an IV Line station, an IUD deployment station, a station for natural delivery and episiotomy repair. The student was observed by the examiner during the procedure in the actual work environment and the results were recorded according to a structured checklist and presented in an objective setting environment. To determine the validity of the checklist, content validity was used and the checklist was approved by faculty members of obstetrics and gynecology. An 85% reliability was calculated based on Cronbach's alpha. Data were analyzed using SPSS version 18 and Chi-square and independent t-test were used at a significance level of 0.05.

 RESULTS

This research was conducted on 60 medical interns and stagers in the department of obstetrics and gynecology of Vali-e-Asr Hospital. The mean age of students was 23.43 ± 5.41 years with a minimum of 18 and a maximum of 32. The DOPS method was used at an average of 20 times by the examiner and the frequency of the procedure for the test was 20 times.

The results of the Chi-square test on evaluation indices at all stations showed that the student's clinical skills were as expected in the majority of cases. The highest expected and above expected were related to communication skills and requesting assistance if needed; while most below expected were related to knowledge of the indicators, anatomy, and procedure’s techniques. (Table 1).
Based on independent t-test, the results showed that there was a significant difference between learners' satisfaction and examiners’ satisfaction (p <0.05).

Discussion:

The aim of this study was to evaluate the clinical skills of interns and stagers using the Direct Observation of Procedural Skills (DOPS) method and satisfaction level of learners and examiners. The results showed that there was a significant difference between the scores of 11 clinical skills of the department of obstetrics and gynecology in the students, and the scores of the skills in all of the evaluation indicators at all stations were in the expected range in the majority of cases. The highest expected and above expected cases were related to communication skills and requesting assistance if needed; while most below expected cases were related to knowledge of the indicators, anatomy, and procedure’s techniques. The most disadvantage mentioned by the student was the technical ability and the most advantage was requesting help if needed.  Overall, the results showed that the majority of students had acceptable grades from the evaluated stations and this method was effective in learning practical skills in the students. In the study of Shah Gheibi et al., Who assessed the interns' skills at Kurdistan University of Medical Sciences with 7 stations based on the DOPS method, DOPS method was effective in students’ learning. (8) The Bazrafkan’s study in Shiraz also showed that the majority of the students obtained a high score from the DOPS test in assessing the practical skills of the lab, which shows that the DOPS test is a useful tool in evaluating practical skills (9) In a study by Habibi et al., the two methods of Mini- CEX and DOPS are more effective in promoting the level of nursing students' skills than the traditional evaluation method (1). Therefore, the use of this method is useful in evaluating practical skills in the department of obstetrics and gynecology. Using this method, the examiner focuses on the important points that need to be evaluated in the skill required. This approach facilitates providing feedback to the experts, because instead of general comment, feedback is based on the actual and objective behaviors. Also, the use of this approach is not only an incentive for learning, but also because the test method and content are directly related to clinical performance, it can lead to a comprehensive learning effort.

Other results the study showed that there was a significant difference between the mean score of satisfaction in learners of 3.5 ± 2.92 and the mean score of satisfaction of examiners in the test was 8.85 ± 0.93. In the study of Akbari et al, regarding the DOPS test in evaluating practical skills in restorative dentistry, 71% of the students were satisfied with the positive effect of this test on learning, which is consistent with the present study. (10) Therefore, the satisfaction degree with the evaluation method was higher in examiners than the learners. Despite the fact that in this assessment the presence of the professors at all stations is necessary and, unlike the OSCE method, it cannot be evaluated by another person, and it is practically difficult and time-consuming for faculty members to perform this test, but due to justice in the evaluation of the students’ practical skills, professors had a good and acceptable satisfaction with this method. One of the limitations of this study was the limited number of moulages in some of the skills and equipment needed for training at the center of clinical skills, and therefore, training required to spend more time and to express some skills in theory on the moulages.

Table 1. Frequency and Significance Distribution of Clinical Skills Assessment by Separate Stations in the Studied Students

Clinical Skills

Unacceptable

Below expected

Moderate

As Expected

Above Expected

Significance

No. (percent)

No. (percent)

No. (percent)

No. (percent)

No. (percent)

No. (percent)

1- Knowledge of indications, anatomy and procedure technique

2(3.3)

3(5)

6(10)

41(68.2)

8(13.3)

0.002

2- Patient Satisfaction

1(1.6)

2(3.3)

8(13.3)

45(75)

4(6.8)

0.001

3. Preparation stage before the procedure

1(1.6)

2(3.3)

12(20)

43(71.8)

2(3.3)

0.021

4- Analgesia and proper sedation

1(1.6)

3(5)

4(6.6)

49(81.8)

3(5)

0.001

5- Observing the sterile conditions

1(1.6)

2(3.3)

6(10)

50(83.5)

1(1.6)

0.001

6. Technical ability (natural delivery)

1(1.6)

2(3.3)

6(10)

47(78.5)

4(6.6)

0.001

7- Requesting help if needed

0(0)

2(3.3)

3(5)

47(78.4)

8(13.3)

0.001

8. Measures after the procedure

1(1.6)

2(3.3)

6(10)

41(68.2)

10(16.6)

0.012

9. Communication skills

0(0)

2(3.3)

8(13.3)

31(51.8)

19(31.6)

0.022

10. Professional behavior and attention to the patient

0(0)

4(6.7)

9(15)

39(65)

8(13.3)

0.013

11. General skill in performing the procedure

1(1.6)

3(5)

5(8.3)

49(81.8)

2(3.3)

0.001

 

Table 2. Comparison of learners and testers satisfaction

Satisfaction Level

Mean

Standard Deviation

Significance

Learners

5.3

2.92

0.002

Examiners

8.85

0.93

Conclusion

Based on the DOPS method, the students' clinical assessment is cost-effective and highly suitable for assessing the practical skills of the obstetrics and gynecology interns, and leads them to direct their education towards clinical skills based on the curriculum of the department, and it can even be a suggested method for changing the educational curriculum of the obstetrics and gynecology internship.

Acknowledgments

The authors need to thank the faculty members of the department of obstetrics and gynecology who helped this project.

 

Ethical Considerations

All tests had no name and surname.

Conflict of interest

There is no conflict of interest between the authors and the Future of Medical Education Journal.

Funds

Funds have been provided by researcher.

 

 

1-Das M. Townesd A, Hasan MY.The views of senior student and young doctors of their training in a skills laboratory. Med educ.1998;32(2):143-149.

2-Amini A, bazaegar M, Hatamy F. the state of clinical competencies of medical student in performing basic clinical procedures at Tabriz University of Medical sciences and Health services. Iranian jornal of medical education. 2001;1(4) :9-16.

3-Pakniat H, Movahed F, Dabagh T, Ghasemi Z. The Effects of Clinical Skills Training on Medical Trainees Performances in Gynecology Ward of Qazvin University of Medical Sciences. Research in Medical Education. 2012; 4 (1) :9-16.

4-Habibi H, Khaghanizade M, Mahmoodi H, Ebadi A, seyedmazhari M. [Comparison of the Effects of Modern Assessment Methods (DOPS and Mini-CEX) with traditional method on Nursing Students' Clinical Skills: A Randomized Trial. Iranian Journal of Medical Education]. 2013; 13 (5) :364-372

5- Abbas Abbaszadeh, Fariba Borhani, Sakineh Sabzevari*, Zohre Eftekhari. [The Assessment Methods and its Relationship to Learning Approaches of Nursing Students in Kerman University of Medical Sciences, Iran. Strides in Development of Medical Education].2013;10(2):142-152

6- Hoseini B, jafarnejad F, Mazlom S, Foroghi pour M, karimi mouneghi H. [Midwifery Students' Satisfaction with Logbook as a Clinical Assessment Means in Mashhad University of Medical Sciences, 2010]. Iranian Journal of Medical Education. 2012; 11 (8) :933-941.

7- Bourbonnais FF, Longford S, Giannatonio L. Development of a clinical evaluation tool for baccalaureate nursing students. Nurse Educ Pract 2008; 8(1): 62-71.

8-Shumwy JM, Harden RM. AMEE Guide No. 25: The assessment of learning outcomes for the competent and

reflective physician. Med Teach 2003; 25(6): 569-584.

9- Bhugra D, Malik A, Brown N. Workplace-based assessment in psychiatry. London: Royal college of

psychiatrists; 2007: 1-13.

10-Amin Z, Chong Y, Khoo H. Direct observation of procedural skills (chapter16). In:Amin Z, Chong Y,

Khoo H. Practical guid to medical student. 1st ed. Singapore: World Scientific Printers CO; 2006: 71-4.

 11-Sahebalzamani M, Farahani H, Jahantigh M. [Validity and Reliability of Direct Observation of Procedural Skills in Evaluating the Clinical Skills of Nursing Students of Zahedan Nursing and Midwifery School]. 3. 2012; 14 (2) :76-81

12-Kogan JR, Holmboe ES, Hauer KE. [Tools for direct observation and assessment of clinical skills of medical trainees: a systematic review]. JAMA. 2009 Sep 23;302(12):1316-26.

13-Naeem N. [Validity, reliability, feasibility, acceptability and educational impact of direct observation of procedural skills (DOPS)]. J Coll Physicians Surg Pak. 2013 Jan;23(1):77-82.

14-Cobb KA, Brown G, Jaarsma DA, Hammond RA. [The educational impact of assessment: A comparison of DOPS and MCQs]. Med Teach. 2013 Jun 28. [Epub ahead of print]

15-Sharemi H, Asgari F.Atraining course on basic gynecological clinical skills and its effect on medical students performance in Guilan Universitiy of medical sciences. Iranian gornal of medical Education. 2011;11(1):111-116.

16-Kebriaei A, Roudbari M. Quality Gap in Educational Services at Zahedan University of Medical Sciences: Students Viewpoints about Current and Optimal Condition. Iranian Journal of Medical Education. 2005; 5 (1) :53-61.

17- Moatarri M, Abdollahzargar SH, Moosavinasab M, et al. [survey on validity and reliability of the test OSCE inevaluating the clinical skills of nursing students at Shiraz nursing college] Persian. Med Res 2007; 31(1): 55-59

18- Farajzadeh Z, Saadatjoo S, Tabiee S, Hosseini M. Reliability and Validity of OSCE in Evaluating Clinical Skills of emergency Medicine Students of Birjand university of Medical Sciences . J Birjand Univ Med Sci.. 2011; 18 (4) :312-319