Quantification of the Content Validity of Written Tests at Mashhad Dental School in 2011-2012

Introduction

Evaluation is a systematic determination of subjects’ merit, value, and significance, which uses criteria governed by a set of standards. Evaluation can help an organization, program, or project to ascertain the degree of achievement or value in regard to its aims and objectives (1).

 The main purpose of evaluation is determining the quality of a program to assess the effects of certain activities in three areas: knowledge, attitude, and behavior. Questionnaire, interview, and observation are three methods of collecting evaluation information (1).

Training evaluation is a continual and systematic process of assessing the worthiness and value of a training course, program, or activity. It is an essential objective of gathering quantitative and qualitative data regarding the effectiveness of training. Various components of training and also continuation, correction, or elimination of the training are determined by the obtained results of evaluation (2).

Training evaluation is the most important and integral part of training, if it is continuously tested and is based on scientific principles. In fact, the obtained results of evaluation should determine whether training assists the organization or program to improve its performance (3).

Today, the high prevalence of dental or oral diseases and socioeconomic complications greatly affect public health. Therefore, we need criteria to assess whether our training is valuable and applicable for the development and modification of dental knowledge and skills (4). There are various methods for training evaluation, and determining the validity and reliability of exam questions is one of the useful methods. The content validity indicates whether the exam questions represent the content and objectives of a specific topic; to wit, a written test has content validity if the questions represent all the objectives and course content (5).

The evaluation of exam questions as a measurement tool for assessing students' learning is one of the main aspects of training evaluation. The examination questions should be carefully evaluated to assess the rate of students’ learning (6).

Academic tests are useful tools for evaluating the knowledge and capabilities of students. Therefore, in order to be certain about the quality of questions, it is essential to assess the reliability and validity of these tests (7). This study aimed to assess the content validity offinal exams at Mashhad Dental School. In addition, the practical purpose of this study was raising the quality of written exams of Mashhad Dental School.

Methods

The present descriptive, cross-sectional study was conducted in order to determine the content validity of final exams at Mashhad Dental School, located in Northeast of Iran, in years 2011-2012. Inclusion criteria were: one of the multiple choice questions from following 11 departments: departments of pediatric dentistry, dental prosthesis, maxillofacial radiology, preiodontics, endodontics, orthodontics, restorative dentistry, maxillofacial surgery, oral diseases and diagnosis, oral health and community dentistry, and pathology. Exclusion criteria were: lack of cooperation in educational groups of Mashhad Dental School.

From all educational groups, one final exam was selected to determine the amount of the content and learning objectives coverage by the tests. For this purpose, one of the faculty members of each group was selected as the representative, and was provided with the required explanations to fill the related questionnaire. In this study, a six-item questionnaire was given to each of the deputy, and was completed by the group members and representatives.

The items included in these questionnaires were as follows:

 1) Determine the percentage of items related to educational objectives.

2) Determine the percentage of educational objectives covered by at least one exam question.

 3) What is the percentage of content validity of the exam?

 4) How many items were unrelated to educational objectives?

5) In which domains of educational objectives, there were no questions associated with educational objectives?

 6) According to the faculties, which domains of education are considered important (based on the number of exam questions)?

The Messick method was used to assess the content validity of Dental School exams. In this study, the percentage of content coverage (CC), and the percentage of content relevance (CR) of each test item were determined by dental school departments. Afterwards, the percentage of content validity (CV) was calculated using Messick method; in fact, content validity is the mean of CC and CR.

In Messick method, a blueprint was used to determine the variables. In this study, the blueprint was assigned to the educational objectives and the syllabus content. Finally, the obtained results were presented to the department of dental school education to promote exam quality. We used frequency distribution table and diagrams in order to assess the content validity of final exams. 

Results

In total, 68 faculty members who were exam developers participated in the current study. The obtained results of each department were reported by the representatives. 

According to the results of this study, the mean percentage of CR and CC was 95.2% and 76.87 in the final exams of all departments, respectively. In addition, the mean percentage of content validity was 86.04 in exams of different groups. The percentages of items related to the educational objectives, which is the percentage of CR of exams of different departments, are indicated in Chart 1. 

The percentages of educational objectives covered by final exam items (the percentage of CC of exams) in different departments are presented in Chart 2. In addition, the CV percentages of exams in different departments are shown in Chart 3.

According to the results, one question item of the exams of both endodontics and oral health and community dentistry departments was unrelated to educational objectives. In addition, three items of both pathology and restorative dentistry departments were unrelated to educational objective. Two, four, and five items of the exams of orthodontics, maxillofacial surgery, and preiodontics departments were unrelated to the educational objectives, respectively. None of the items in the final exams of pediatric dentistry, dental prosthesis, maxillofacial radiology, and oral diseases were unrelated to the objectives.

Finally, we determined the most important educational domains in different departments of Mashhad Dental School. The percentages of questions without CC of educational domains are demonstrated in Chart 4.

 The radiographic and clinical diagnosis of complications related to dental trauma to primary teeth on the permanent dentition, selecting the type of spacer, treatment of luxation injuries, and preparation techniques for managing avulsed tooth were the most significant educational domains of the department of pediatric dentistry. The most important learning domains of the department of endodontics included root canal preparation techniques, and radiography in endodontic treatment and internal anatomy of the teeth. Oral public health, public health, socioeconomic consequences of tooth decay, and types of epidemiological studies were the most important educational domains of the department of oral health and community dentistry.

Also, the most significant educational domains of the department of maxillofacial surgery included flap surgery, pre-prosthetic surgery, and familiarity with dental equipments. The embryological origin of the formation of different facial components, developmental patterns of cranial base, complex nasolacrimal duct obstructions¸ and mandible, and the mechanism of tooth development (before and after the development in the mouth) were the most important educational domains of the department of orthodontics.

The important educational domains of the department of pathology consisted of chronic mucocutaneous candidiasis, developmental disorders, and microbial diseases. In addition, the gingiva, plaque and calculus, and the etiology of periodontal diseases were the important educational domains of the department of preiodontics.

The important educational domains of the department of dental prosthesis included retention in partial dentures and the related factors, forms of indirect retainers and the related factors, and components of partial dentures. The most important educational domains of the department of maxillofacial radiology were knowledge of the factors affecting image quality and film processing, and factors affecting X-ray and tissue sensitivity to radiation. Also, complex amalgam restorations, examination, diagnosis and treatment planning, and pulp-protection materials were the most important educational domains of the department of restorative dentistry. Finally, laboratory tests, red and white lesions, and exophytic lesions were the most important educational domains of the department of oral diseases.

Discussion

The results of this study showed that the mean percentage of CR and CC of final exams was 95.2% and 76.87% in all educational groups, respectively. Also, the mean of CV of final exams was 86.04% in all groups.

The results of the current study were consistent with the study of Kazemi and Ehsanpour, which assessed the content validity of exam questions of theoretical courses of midwifery. In addition, the degree of compliance between exam questions and the level of students’ learning was assessed in the mentioned study. The results indicated that the content validity and face validity of exam questions were 92.38% and 84.45%, respectively. However, the level of students’ learning was not consistent with the expected level (8).

Also, the results of the present study were in consistence with studies of Edhami et al. (9), Heravi et al. (10), and Hasnzadeh et al. (11).

Jahani Shorab in his study aimed to assess the reliability and validity of assessment tools of the final examination of the department of nursing and midwifery. The results of that study indicated that the assessment tools of comprehensive examination were reliable tools for evaluating students at the time of graduation (12).

The obtained results of this study indicated that there was a large difference between the percentage of CR and CC. The percentage of CR ranged between 80% and 100%, though the percentage of CC ranged between 45.71% and 92.3%. Therefore, CC plays a more significant role in comparison with CR in determining the percentage of content validity.

Also, other results of this study showed that there was no relationship between the number of faculty members and number of educational objectives. Also, there was no relationship between the gender of faculty members and the number of educational objectives.

Consistence with our study, Kazemi and Ehsanpour (8) and Jandaghi (13) used the Messick method to assess content validity. However, Hsanzade et al. (11) and Jahani Shorab et al. (12) used the Lawshe method for their assessment of content validity.

Najar and Abedi in their study assessed the rate of awareness of faculty members regarding measurement methods of reliability and validity of exam questions. The results indicated that 80% of faculty members had sufficient knowledge about content validity to determine the validity of exam questions. However, a small percentage of subjects had sufficient knowledge about the methods of assessing reliability and validity of exam questions (14).

Shaban and Ramezani Badr evaluated the content validity of exam questions of nursing and midwifery of Tehran University of Medical Sciences. According to their study, the educational pamphlets to faculty members have a considerable effect on improving the designed questions (15).

This study concluded that the mean percentage of CR and CC of final exams was good (95.2% and 76.87%) in all educational groups. Regarding the validity of the exams, evaluating the content aspect and the supervision of the standard designation of questions by the EDO could lead to an improvement in the quality of the examinations and their validity, which was suggested to be generalized for all other programmed exams. Moreover, teachers' awareness of the importance of content validity and calculation methods are very effective in designing efficient exams with greater sensitivity.