Real Time Electronic Formative Assessment in Medical Education and Its Impact on Motivational Beliefs and Self-regulation Strategies

Document Type : Original Article

Authors

1 Department of Instructional Technology, Faculty of Psychology and Educational Sciences, Allameh Tabataba’i University, Tehran, Iran

2 Department of Medical Informatics, Faculty of Medicine, Mashhad University of Medical Sciences, Mashhad, Iran

3 Department of Assessment and Measurement, Faculty of Psychology and Educational Sciences, Allameh Tabataba’i University, Tehran, Iran

Abstract

Background and Objectives: The aim of this study was to investigate the effect of real-time electronic formative assessment-based medical education on learners’ motivational beliefs and self-regulation strategies.
Methods: This randomized clinical trial was conducted on 323 students studying at the Medical School of Mashhad University of Medical Sciences, Mashhad, Iran, during the first semester of the academic year of 2017-2018 using a posttest control group design. The investigation of motivational beliefs and self-regulation strategies was accomplished using the Motivated Strategies for Learning Questionnaire. The research hypotheses were studied by means of independent t-test and multivariate analysis of variance.
Results: According to the results, there was a significant difference between the control and intervention groups in terms of motivational strategy subscales, namely self-efficacy, intrinsic value, test anxiety, and self-regulation (P0.05).
Conclusion: As the findings indicated, the use of real-time electronic formative assessment in the educational environment can enhance students’ motivational beliefs and self-regulation strategies in medical education.
 

Keywords

Main Subjects


 

Assessment is one of the important steps in any teaching-learning process [1, 2]. Brown and Knight consider assessment as the heart of student learning experience [3]. In new learning theories, classroom assessment is recognized as an important element in both teaching and learning processes [4, 5].According to Butler and Mcmunn (2006), assessment can be distinguished under three types of diagnostic, formative, and summative, which date back to Scriven (1967) who first made the distinction [6]. 

Formative assessment is defined as a continuous process of monitoring the learners' progress in obtaining knowledge [7]. In another definition, this kind of assessment is defined as "the process of seeking and interpreting evidence for the use by learners and teachers to decide where the learners are in their learning, where they need to go, and how best to get there" [8]. New learning environments are targeted toward the establishment and development of a deep and meaningful learning approach in students [9, 10]. In such learning environments, students have an active, constructive, self-regulated, targeted, and collaborative role, who engage themselves in these environments with the goal of self-promotion [11].

Formative assessment has a positive effect on the learner empowerment and learning self-regulation by helping students to find their strengths and weaknesses [12, 13]. On the other hand, the reduction of learning motivation in medical students is a challenge that has been considered in the recent years. Formative assessment can be used as an option for the enhancement of learner's motivation.

According to some evidence, the use of formative assessment, along with the summative assessment, can improve the quality of education, increase the learning motivation, and improve self-regulation learning in students [13]. In the same vein, Faber et al. reported that the use of electronic formative assessment tools exerts a positive impact on learners' academic achievement and motivation [14].

The advancement of technology has resulted in the establishment of many educational opportunities in the learning environments. One of these tools is the real-time electronic system in the classroom [15]. These technologies facilitate recurring opportunities for students to practice. Electronic formative assessment cannot only involve and motivate students, but also help them monitor their progress and learning speed [16]. These systems enjoy many merits, such as improvement of attention, enhancement of knowledge acquisition, possibility of performing secret surveys, tracking of individual responses, instant display of the responses of questions, creation of an interactive and entertaining learning environment, collection of information for reporting, and implementation of analysis [17].

It should be also noted that such assessments motivate the learners to reflect, discuss, and participate in the learning process [18]. One of the major challenges in the field of medical education is that in large classes, it is difficult to achieve high and acceptable lecturing standards and present materials while involving the students and increasing their participation and academic engagement. Individual response technology encourages active learning [19] and leads to the activation of higher levels of cognition in students.

It should be noted that motivation is one of the most important determinants of learning quality and success. Accordingly, the lack of this construct can well explain such questions as why professors sometimes encounter with discouraged students or learners who have lost their interest or have abandoned their studies and activities, as well as why sometimes the students feel weak or abandoned [20]. One of the objectives of the university environment is to encourage the students to engage in a social, sustainable, and non-threatening environment [21]

The medical education researchers should give special attention to the concept of motivation [20]. However, the mechanisms through which learners adjust their motivation and learning strategies are not fully understood yet [22]. According to Borman and Sleigh, learners can be significantly engaged in the learning environments by means of non-summative marking systems [23]. Electronic methods for feedback presentation can be also developed automatically and continuously to support learners' engagement in learning [15].

According to the literature, real-time electronic formative assessment facilitates the learners’ active participation in the learning environment and allow them to receive immediate feedback [21]. The technique investigated in this study specifically modifies the role of students from a passive position to the condition in which students take the responsibility of their learning process and seek to improve the process through active participation in the learning environment. The present study aimed to find a strategy to support and guide learners in real time [24].

This study also attempted to find out how electronic formative assessment could be provocative for learners. To this end, a learning environment was designed based on real-time electronic formative assessment that involved gamification elements as motivational aspects in the process of teaching and learning. Gamification refers to the mechanical and artistic application of ideas and aesthetic components of games (e.g., context, immediate feedback, competition, stages, achievements, and points) to engage the learners in the problem and motivational activities and promote their learning and problem solving ability [25]. Generally, the purpose of the present study was to investigate the effect of real-time electronic formative assessment-based medical education on students’ motivational beliefs and self-regulation strategies.

Materials and methods                                                                                                        

This randomized clinical trial was conducted on 323 students studying at the Medical School of Mashhad University of Medical Sciences, Mashhad, Iran, during the first semester of the academic year of 2017-2018 using a posttest control group design. The research hypotheses were as follows:

a) Students exposed to real-time electronic formative assessment-based learning model have a higher motivational belief score, compared to the non-exposed students.

b) Students exposed to the real-time electronic formative assessment-based learning model have a higher self-regulation strategies score, compared to the non-exposed students.

Study population and sampling

The study population corresponded to a group of 323 students selected from 1,620 MD students studying at Mashhad University of Medical Sciences using convenience sampling technique. The participants were randomly divided into two groups of intervention (n=159) and control (n=164).

Research process

Given that the difficulty of the educational content of each lesson could disturb the results of the study, there was a need to homogenize this variable. Regarding this, based on the opinions of the relevant experts and professors, the lessons of histology and trunk anatomy, which were similar in terms of difficulty level, were selected from the basic science courses for general medical students. The sessions were divided into two groups of control and intervention.

After the determination of the control and intervention groups, in the intervention sessions, real-time electronic formative assessment was adopted during the teaching process. In this regard, during each session, the instructor asked questions regarding the educational content taught at the same session in the form of PowerPoint slides. The students used keypads to answer the questions.

Based on the rate of the correct answers, the professor instantly (in the real time) decided on continuing the discussion or re-explaining the subject to resolve the gap between what is gained by the students and what they should know. In the control group, the sessions were implemented using the routine educational approach. At the end of the course, the two groups filled out the Motivated Strategies for Learning Questionnaire (MSLQ) developed by Pintrich and DeGroot (1990) [26]. This questionnaire measures the students' motivational beliefs and self-regulation strategies. The MSLQ consists of two main sections, namely motivational beliefs (including three subscales of self-efficacy, intrinsic value, and test anxiety) and self-regulation strategies (including two subscales of cognitive and self-regulation strategies).

Statistical analysis

Regarding the fact that the education was performed by two different lecturers in the control and intervention groups, the educator factor was in interaction with educational method; therefore, the design of the study could be considered with two independent variables.Therefore, a preliminary analysis was carried out separately, considering the roles of the teaching method and educator as independent variables in the subscales of motivational strategies for learning to use the independent t-test.

The results indicated that the intervention group exposed to the real-time electronic formative assessment-based learning showed higher mean scores in self-efficacy, intrinsic value, and self-regulation than the control group (P<0.05). In addition, the intervention group had a lower test anxiety mean score, compared to the control group (P<0.05). The only subscale that was not significantly different between the two groups was cognitive strategies (P>0.05). However, the investigation of educator factor as an independent variable revealed no significant difference between the two lecturers in terms of motivational strategies for learning. The performance of the two groups was comparable regardless of the grouping factor (i.e., lecturer) (P>0.05). Therefore, due to observing no difference in the subscales when considering the educator factor, this factor was ignored in all analyses of motivational strategies for learning subscales.

Since the five components (i.e., self-efficacy, test anxiety, intrinsic value, cognitive strategies, and self-regulation strategies) under study were aggregated in one scale, the separate analysis of the major two components (i.e., motivational beliefs and self-regulation strategies) was avoided, and they were analyzed collectively. Consequently, the research hypotheses were tested using the multivariate analysis of variance (MANOVA) with five dependent variables and one independent variable, namely teaching method.

Results

The distribution of the data related to motivational strategies for learning was tested using skewness and kurtosis. In addition, the Kolmogorov-Smirnov test was employed to investigate the normality of the data.

Table 1. Descriptive statistics and normality of motivational strategies for learning

Statistics

 

 

Subscale

 

 

Range

Minimum

Maximum

Mean

Standard error of mean

Standard deviation

Variance

Skewness

Standard error of skewness

Kurtosis

Standard error

Kolmogorov-Smirnov

P-value

Self-efficacy

28

17

45

33.89

0.30

4.99

24.99

-0.32

0.148

0.34

0.29

1.86

0.002

Intrinsic value

34

26

60

47.34

0.36

5.72

32.76

-0.46

0.153

0.99

0.30

1.22

0.10

Test anxiety

24

6

30

17.63

0.29

4.84

23.44

0.12

0.148

-0.43

0.29

1.43

0.04

Cognitive strategies

38

42

80

60.62

0.41

6.64

44.11

0

0.152

0.83

0.30

1.33

0.06

Self-regulation

12

8

20

14.98

0.14

2.23

4.99

-0.37

0.151

0.31

0.30

2.35

0.0001

 

 

Table 1 tabulates the descriptive statistics and normality of the data related to motivational strategies for learning. As indicated in this table, the dependent variable had a normal distribution in the subscales of motivational strategies for learning. It should be noted that the variables whose normality was not confirmed by the Kolmogorov-Smirnov test were evaluated using the skewness and kurtosis. Given the fact that there were five subscales in motivational strategies for learning and given the interdependence of these subscales, MANOVA was used to study the hypotheses considering the educational method as the independent variable.

 

Table 2. Effect of real-time electronic formative assessment-based learning environment on the subscales of motivational strategies for learning

 

Subscale

Research groups

F

P-value

Intervention group

Control group

Mean±SD

Mean±SD

Self-efficacy

35.01±4.48

32.80±5.25

12.57

0.0001

Intrinsic value

48.76±5.26

45.87±5.83

16.57

0.0001

Test anxiety

16.74±4.76

18.50±4.78

8.30

0.004

Cognitive strategies

61.06±6.36

60.19±6.92

0.43

0.51

Self-regulation

15.29±2.11

14.66±2.32

5.47

0.02

Table 2 presents the results of MANOVA. The results indicated a significant difference between the intervention and control groups in terms of the mean scores of self-efficacy, intrinsic value, test anxiety, and self-regulation strategies (P<0.05). In this regard, the intervention group had higher mean scores in these subscales, compared to the control group. The results of MANOVA also revealed that the motivational strategies in the students exposed to the real-time electronic formative assessment-based learning were at a more favorable level, compared to those in the students, who were not subjected to this educational model with 95% confidence interval. However, no significant difference was observed between the two groups regarding the subscale of cognitive strategies (P>0.05).

Discussion and Conclusion

The findings of this study revealed a significant difference between the intervention and control groups regarding the mean self-efficacy, intrinsic value, test anxiety, and self-regulation. In other words, the intervention group, exposed to a real-time electronic formative assessment-based learning environment, showed more favorable outcomes in terms of these subscales, compared to the control group. However, no significant difference was observed between the two groups considering the mean score of cognitive strategies. This finding can be due to the fact that the induction of a change in the cognitive strategies of the students requires implementing the intervention for a longer period of time.

In general, the findings of the present study are in line with those obtained by Ghazi and Henshaw (1998), Black et al. (2003), Nazari and Osareh (2011), Weurlander et al. (2012), Ludvigsen et al. (2015), Faber et al. (2017), and Georgoff et al. (2018) [14, 27-32]. Clark (2012), investigating formative assessment and its impact on self-regulation learning, introduced this method as a part of learning activities that improves self-regulation learning [33]. Likewise, Pilli and Aksu (2013) found that electronic formative assessment tools exert a positive effect on learners' attitudes [34].

On the other hand, our findings are inconsistent with the results obtained by Muis et al. (2015) reporting negative impacts of electronic formative assessment on the motivation of five-year-old children [35]. A possible explanation for this finding can be related to the effects of negative feedback on learners’ motivation. Hunsu et al. (2016) also indicated the positive effects of audience response system on non-cognitive learning outcomes, such as student engagement, participation, and interest [36]

In general, studies have shown that the use of formative assessment in the implementation of educational courses facilitates the learner to turn from a passive learner to an active one who is responsible for his/her own learning. It seems that the use of electronic formative assessment in classrooms can enhance students' motivation for learning and strengthen self-regulation learning in them. As the literature indicated, the use of formative assessment, along with summative assessment, leads to the improvement of education quality, enhancement of motivation for learning, and reinforcement of self-regulation learning in students.

 

Acknowledgements

This study was derived from a PhD thesis titled, "Design and Development of Learning Environment Model Based on Real Time Electronic Formative Assessment in Medical Education and Its Impact on Learning and Motivated Strategies for Learning". Hereby, we extend our gratitude to the Mashhad University of Medical Sciences for their support.

Ethical considerations

In line with the research ethics principles, informed consent was obtained from all participants. Furthermore, the participants were ensured about the anonymity terms; accordingly, their scores were used through coding. The present study was ethically approved by the Research Committee.

Conflicts of interest

There are no conflicts of interest in this study.

References
[1]. Vonderwell, S., Liang, X. and Alderman, K. (2007). Asynchronous Discussion and Assessment in online Learning. Journal of Research on Technology in Education; Spring 2007; 39, 3; ProQuest Education Journals. P 309.
[2]. Nadjafi-Semnani, M., Nadjafi-Semnani, A., Nadjafi-Semnani, F., Bijari, B., Mohammadi, Y. and Ghanbarzadeh, N. (2017). Assessing the clinical skills of interns and stagers in the department of obstetrics and gynecology using Direct Observation of Procedural Skills (DOPS) method and satisfaction level of learners and examiners. Future of Medical Education Journal. Article 3, Volume 7, Issue 3, September 2017: 14-17.
[3]. Brown, S. and Knight, P. (1994). Assessing Learners in Higher Education, London: Kogan Page.
[4]. Scouller, K. (1998). The influence of assessment method on students’ learning approaches: Multiple choice question examination versus assignment essay. Higher Education, 35: 453–472.
[5]. Gibbs, G. (1999). Using assessment strategically to change the way students learn. In S. Brown and A. Glasner (Eds.), Assessment matters in higher education: Choosing and using diverse approaches (pp.41–53). Buckingham: SRHE and Open University Press.
[6]. Butler, S.M. and Mcmunn, N.D. (2006). A Teacher Guide to Classroom Assessment. San Francisco: Jossy-Bass co Press; 2-3.
[7]. Hsu, J.L., Chou, H.W. and Chang, H.H. (2010). EduMiner: Using text mining for automatic formative assessment, Expert Systems with Applications.
[8]. Assessment Reform Group (ARG). (2002). Assessment for learning: 10 principles, Cambridge,UK: University of Cambridge, School of Education.
[9]. Fyrenius, A., Bergdahl, B., and Sile´n, C. (2005). Lectures in problem-based learning: why, when and how? An example of interactive lecturing that stimulates meaningful learning. Medical Teacher, 27(1):61–65.
[10]. Karimi Moonaghi, H. and Bagheri, M. (2017). Jigsaw: A good student-centered method in medical education. Future of Medical Education Journal. Article 7, Volume 7, Issue 1, March 2017: 35-40.
[11]. De Corte, E. (2000). Marrying theory building and the improvement of school practice: A permanent challenge for instructional psychology. Learning and Instruction, 10(3): 249–266.
[12]. Nicol, D.J. (2006). Formative assessment and selfregulated learning: A model and seven principles of good feedback practice. Studies in Higher Education (2006), Vol 31(2): 199-218.
[13]. Cauley, K.M. and McMillan, J.H. (2010). Formtive assessment techniques to support student motivation and achievement. Formtive assessment techniques to support student motivation and achievement. 2010;83(1): 1-6.
[14]. Faber, M.J., Luyten, H. and Visscher, A.J. (2017). The effects of a digital formative assessment tool on mathematics achievement and student motivation: Results of a randomized experiment. Computers and Education 106: 83-96.
[15]. Bull, J. and McKenna, C. (2004). Blueprint for computer-assisted assessment (London, Routledge-Falmer).
[16]. Jordan, S. (2013). E-assessment: Past, present and future. The Higher Education Academy, NDIR, Vol 9, Issue 1: 87-106.
[17]. Brewer, C.A. (2004). Near Real-Time Assessment of Student Learning and Understanding in Biology Courses. BioScience. Vol. 54 No. 11: 1034-1039.
[18]. Tenorio, T., Bittencourt, I., Isotani, S., Pedro, A. and Ospina, P. (2016). A gamified peer assessment model for on-line learning environments in a competitive context. Computers in Human Behavior 64(2016): 247-263.
[19]. Gan, C.L. and Balakrishnan, V. (2017). Enhancing classroom interaction via IMMAP-An Interactive Mobile Messaging App. Telematics and Informatics 34: 230–243.
[20]. Pelaccia, T., and Viau, R. (2017). Motivation in medical education. Medical Teacher. Volume 39, Issue 2: 136-140.
[21]. Heden, L. and Ahlstrom, L. (2016). Individual response technology to promote active learning within the caring sciences: An experimental research study. Nurse Education Today 36 (2016) 202–206: 202-206.
[22]. Zhang, Y., Lin, C., Zhang, D., and Choi, Y. (2017). Motivation, strategy, and English as a foreign language vocabulary learning: A structural equation modeling study. British Journal of Educational Psychology. Volume 87, Issue 1: 57-74.
[23]. Borman, D. and Sleigh, A. (2011). An evaluation of the use of interactive approaches and integrated on-line resources. Teaching Mathematics and its Applications, 30(4): 166–177.
[24]. Pedro, M.S. (2013). Real-time Assessment, Prediction, and Scaffolding of Middle School Students’ Data Collection Skills within Physical Science Simulations. A Dissertation Submitted to the PhD committee in Partial Fulfillment of the Requirements for the Dissertation Proposal of Doctor of Philosophy In Learning Sciences and Technologies.
[25]. Kapp, K. M. (2012). The gamification of learning and instruction: Game-based methods and strategies for training and education. John Wiley and Sons.
[26]. Pintrich, R.R. and DeGroot, E.V. (1990). Motivational and self-regulated learning components of classroom academic performance. Journal of Educational Psychology, 82: 33-40.
[27]. Ghazi, F. and Henshaw, L. (1998). How to keep student nurses motivated. Nursing Standard 13 (8): 43–48.
[28]. Black, P., Harrison, C., Lee, C., Marshall, B. and William, D. (2003). Assessment for learning: putting it into practice. United Kingdom, McGraw-Hill: Open University Press; 2003.
[29]. Nazari, A. and Osareh, F. (2011). Perspective of Shahrkod High School Students about the Factors Encouraging or Inhibiting the Free Study. National Studies on Librarianship and Information Organization 2011; 87:57-73. [Persian]
[30]. Weurlander, M., Söderberg, M., Scheja, M., Hult, H. and Wernerson, A. (2012). Exploring formative assessment as a tool for learning: students’ experiences of different methods of formative assessment. Assessment and Evaluation in Higher Education, 37(6): 747–760.
[31]. Ludvigsen, K., Krumsvik, R. and Furnes, B. (2015). Creating formative feedback spaces in large lectures. Computers and Education 88 (2015): 48-63.
[32]. Georgoff, P.E. et al. (2018). Evaluating the performance of the Minute Feedback System: A web-based feedback tool for medical students. The American Journal of Surgery 215 (2018): 293-297.
[33]. Clark, I. (2012). Formative Assessment: Assessment Is for Self-regulated Learning. Educational Psychology Review, 24(2): 205–249.
[34]. Pilli, O. and Aksu, M. (2013). The effects of computer-assested instruction on the achievement, attitudes and retention of fourth grade mathematics students in North Cyprus. Computer and Education, 62: 62-71.
[35]. Muis, K.R., Ranellucci, J., Trevors, G. and Duffy, M.C. (2015). The effects of technology-mediated immediate feedback on kindergarten students' attitudes, emotions, engagement and learning outcomes during literacy skills development. Learning and Instruction, 38: 1-13.
[36]. Hunsu, N.J., Adesope, O. and Bayly, D.J. (2016). A meta-analysis of the effects of audience response systems (clicker-based technologies) on cognition and affect. Computer and Education, 94: 102-119.