Assessment of Construct Validity Faculty Members’ Evaluation Questionnaire by Students in Health School of Mashhad University of Medical Sciences

Introduction:

One of the major goals of universities is to train specialist staffs. One of the important pillars of this process is university lecturers, who educate students and prepare them for the work market. The study of university professors’ characteristics and merits is inevitable owing to their high duty. The evaluation system of faculty members at universities of medical sciences is more important, as faculty members and their graduates are responsible for humans’ health and life (1, 2).

Evaluation can identify professors’ strengths and weaknesses and leads the system from static to dynamic with high quality. In addition, it attempts to strengthen the educational system by providing appropriate strategies (3). The assessment of professors’ educational environment is one of the most complex types of assessment (4). Universities generally use different methods to evaluate their professors; however, their ultimate goal is to improve the teaching method, enhance and promote effective activities, and eliminate ineffective and undesirable methods (5). Professors' evaluation is one of the most complex types of evaluation in academic environments, since the characteristics of the community to be evaluated are unique and their tasks are diverse. The process of evaluating professors can be influenced by various parameters such as executive responsibilities, characteristics of the lesson presented, conditions and facilities of the university and class, and economic and livelihood issues; therefore, considering each of these cases seems to be impossible in practice (6). Thus far, several methods, including evaluation by faculty colleagues, evaluation by students, evaluation by educational experts, and evaluation by checking the educational documents have been proposed to evaluate professors (7). The most common method used by Iranian universities is the assessment of faculties by their students.

Researchers have employed different parameters to evaluate faculties by students; some of these parameters are students’ educational level, level of difficulty or simplicity of the class, number of students in the class, students’ personality, teachers’ academic level, students' semester and their expected and actual grades, philosophy of education, teachers’ demographic characteristics such as gender, age, educational level, background and teaching experience, as well as physical and social attractions  (8-10).

Considering that the main group of activities of teaching staff is related to students; therefore, it is important to check the tools used in this section. Although many important parameters cannot be evaluated in practice, the low credibility of the measurement tool can be detrimental to the process. Based on the present knowledge, a standard form for student evaluation has not been designed so far, and universities and faculties have designed and used a separate form according to their needs.

The validity and reliability of the questionnaire used in the evaluation of professors are highly important to achieve credible results. Thus far, various methods have been proposed to assess the validity of the questionnaires, including content validity, construct validity, and criterion validity. In this study, we aim to investigate the validity of the EFSQ used at the School of Health of MUMS using confirmatory factor analysis (CFA).

Methods:

The present research is a descriptive-analytic cross-sectional study. The statistical population consists of students at the School of Health of MUMS in the first semester of 2018-2019. All the students who completed the EFSQ were considered. The inclusion criteria were studying at the School of Health of MUMS in 2018, and the exclusion criteria were incomplete completion of the questionnaire. We consider the ethical issues such as confidentiality of information, and non-prejudice of researchers in interpreting the results of the study. The data-gathering tool in this study was the EFSQ at the School of Health of MUMS, which was completed by students on their portal. The EFSQ consists of 12 questions with a Likert scale of 10 points. The range of scores is between 1 and 10 in which higher scores are a sign of the better quality of teacher training. The following table lists the questions in four sections, including education, discipline, ethics, and management.

Table 1. The EFSQ at the School of Health of MUMS

This study used SPSS and AMOS software programs to analyze the data in descriptive and analytical formats. Also charts and tables were employed to describe the data and evaluate the structural validity by exploratory and confirmatory factor analysis. The normality of data was confirmed using skewness and kurtosis indices and Shapiro-Wilk test. The Mahalanobis distance was used to evaluate outlier data. The significant level was considered 5%.

Results

The rate of responding to the EFSQ was 92% (2044 out of 2219). The average total score of the EFSQ was 8.77 ± 1.30 out of 10. Table 2 presents the descriptive statistics for the EFSQ dimensions.

Table 2. Descriptive statistics of scores in the EFSQ

Figure 1 which shows the CFA results using standard factor loading values and goodness of fit indices, indicates the weak fit of this model. Therefore, the current EFSQ used to evaluate faculty members is inappropriate and needs to be reviewed. Given that the minimum sample size required in a CFA is 10 to 20 cases per parameter, the sample size used in this study is sufficient. In addition, the adequacy of the sample size after fitting the CFA was also confirmed using the HOLTER test.

To achieve the proper structure, the data was divided into two parts. In the first part, to obtain the proper structure, exploratory factor analysis (EFA) was performed, and in the second part, the proposed structure obtained from the previous step was evaluated by using a CFA. The value of the KMO index (0.805) indicates the adequacy of the sample size for this analysis. In the EFA with Quartimax rotation, the model with three factors was evaluated as the best model. Table 3 presents the factor loading values.

(Table 3. EFA results on 50% of the data (Train Data

Figure 2 shows the results of the CFA of the proposed model on 50% of the data (Test Data), indicating the good fit of this model.

Table 4 presents the goodness of fit indices for the CFA, indicating that the model is proper.

Table 4. Goodness of fit indices for the CFA on 50% of the data (Test Data)

The values of various internal consistency indices, including Cronbach's alpha and Composite Reliability, were almost equal to and greater than 0.7, confirming the internal consistency of the model.

(Table 5. Indices of internal consistency assessment on 50% of the data (Test Data

 Discussion

One of the most controversial issues in the teaching process is the evaluation of the educational system, particularly the evaluation of professors (12). Undoubtedly, the results of accurate evaluation illustrating the strengths and weaknesses provide a basis for decision-making on educational programs and arrange a framework to improve the quality of university education. From the students’ viewpoint, one of the most important methods to evaluate the faculty members to show the professors’ performance among the most important community of their target is students. Although the aim of the evaluation is to promote the education, as well as growth and development of the faculty members; however, in many universities, it is merely a criterion for promotion of professors (3, 13). The aim of this study was to determine the proper structure of the EFSQ at the School of Health of MUMS. The results of this study indicated that the questionnaire with four dimensions was not suitable, and the most appropriate structure for these questions was the form with three dimensions. Based on the CFA and EFA results, the questions 1, 2, 11 and 12 determined the education scope. 

Since questions 3, 4, and 5 were in one scope, the title of commitment was proposed for this dimension. Although the set of questions 6 to 10 was in one area, it seemed that the sixth question was ambiguous. Regarding the lack of a precise definition for the statement "professional style", this study suggested the removal or modification of this question. Furthermore, the present researchers thought that the title of discipline was appropriate for this domain. Previous studies have indicated that in addition to a teacher’s academic and practical skills, the attitudinal, moral, as well as personality traits are also factors which introduce a teacher as an appropriate professional pattern (14). In addition, given that the eleventh question had the lowest standard factor loading (less than 0.5), it was suggested that this question be removed and replaced with another appropriate question.

The third question of the EFSQ was concerning the correct and fair assessment by the professors; however, it was noticeable that the evaluation of students by teachers has been generally conducted at the end of the semester, and the EFSQ was completed before the final exam; therefore, the students' answer to this question was doubtful; so the deletion of this question was suggested (15).

However, some studies have reported that the EFSQ scores are associated with students’ expected score so that if students expect a high score, they will give a high score to the teacher and vice versa (16).

In one study, Asgharpour et al. examined the factors affecting the student's evaluation of the professor from faculty members and students’ perspective at the Faculty of Nursing and Midwifery of Tehran University of Medical Sciences. In this study, professors and students agreed and approved that 18 qualities in the field of individual-personality traits of the teacher and 32 items relating to evaluation of the professor were high value items (17). Comparing the questions of the present study with the above high value items shows that many of them are collectively referred to one question; for example, Question 2 of this study "Teaching in accordance with educational goals and time defined in curriculum" covers many issues; however, other issues such as the use of visual and audiovisual equipment and attention to students’ understanding determined in the study of Ali Asgharpur et al., were not available in the evaluation form of the School of Health of MUMS.

The results of this study demonstrated that some students used the same pattern to answer questions, which could be due to lack of motivation to participate in the evaluation process. This result is consistent with the results of studies that do not support teachers’ evaluation by students. These studies stated some reasons for not approving this evaluation method. Some of these reasons are lack of justification of the students before completing the form, lack of responsibility of students, lack of feedback to the professors and consequently, their impact on the teaching method, as well as lack of influence on promoting the professors (8, 15, 18). Voluntary participation, considering incentive privileges, and qualitative evaluations can be a step toward solving this problem.

The EFVS is a 10-option Likert spectrum that can cause confusion and biased results. Previous studies have shown that the five-choice questions in Iran are the most appropriate ones (19); however, the present study suggests that the number of options be reduced to five.

Student assessment scores have been used in many studies to examine the effectiveness of education, which is based on the belief that students learn more from professors with higher grades (20, 21). However, a meta-analysis study indicated that student grades should not be considered a factor associated with the effectiveness of professors’ teachings (22).

Based on the factor analysis results, the instrument validity was not approved. It is necessary to modify the evaluation process, for example, by voluntary participation in the evaluation. In general, due to mismatch of the EFSQ at the School of Health of MUMS with the EFSQ in other universities, the present researchers recommend that the questions be revised and the content validity be evaluated through the panel of experts.

Ethical considerations 

Ethical issues (Including plagiarism, informed consent, misconduct, data fabrication and/or falsification, double publication and/or submission, redundancy, etc.) have been completely observed by the authors.

Acknowledgement

We thank all the students who participated in this study for generously helping us in this research. The authors would like to thank all the members of the Educational Development Office (EDO) for their valuable and constructive comments.

 Financial Support: The present study was financially supported by the Research Council of Mashhad University of Medical Sciences with code number: 971908.

Conflict of interest: None declared.