Influence of Assessment Method Selection in Studying and Learning Approaches: Is It Necessary to Change Assessment Style?

Document Type: Original Article

Authors

1 Physiology Research Center, School of Health, Kerman University of Medical Sciences. Kerman, IRAN Department of Nutrition, School of Health, Kerman University of Medical Sciences. Kerman, IRAN

2 Research Center for modeling in health, Institute Futures Studies in Health and Biostatistic and epidemiology Department, School of Health, Kerman University of Medical Sciences. Kerman, IRAN

3 Physiology Research Center, School of Health, Kerman University of Medical Sciences. Kerman, IRAN

Abstract

Background: It is important to learn how to study for different examinations. The objective of the current study is to explore whether the assessment method selection would significantly affect the studying and learning approaches of students.
Methods: This descriptive-analytical research consisted of 191 first-year undergraduate nursing students from three nursing schools and was conducted during two interval semesters. All of students were given examinations with half of multiple-choice questions (MCQs) and half of short-answer questions (SAQs) in mid and final term examinations. A structured 12-item questionnaire was designed based on a modified 5-point Likert scale in outline of visual analogue scale. The questionnaire with Cronbach’s alpha of at 0.814 sought information on student’s studying and learning approaches.
Results: The students significantly rated agreement with queries of short-answer assessment part of questionnaire higher than in multiple-choice assessment part. Female students performed significantly better in the learning outcomes in SAQ and MCQ examinations grades and total grade of both examinations than male students who participated in the study (p= 0.001, 0.001 and 0.030, respectively). Neither lower-quartile nor higher-quartile of education promotion exposed significance difference on total score of studying and learning approaches questionnaires. Responses frequency to studying and learning approaches questionnaire revealed that most students selected short-answer assessment method. 
Conclusion: Therefore, the findings revealed that assessment method may shape and improve students’ studying and learning approaches. Short answer question is hypothesized to enhance the development of deep learning.

Keywords


INTRODUCTION

In education, examinations are almost exclusively used as assessment tools, thus teachers assess the efficacy of their curriculum and students’ learning outcomes in the course to assign grades (1).

Currently, medical school examinations comprise a range of different assessments, both written and performance-based, offering an opportunity to compare performance on different formats (2). The multiple-choice question (MCQ) type of tests is commonly used in assessment of medical knowledge acquisition (3). Objective testing, specifically MCQ is one of the approaches that may diversify the assessment approach in education. Significant commitment is required to prepare MCQ test items and examination formats that are reliable and consistent with curriculum objectives. Appropriately constructed MCQ examinations are efficient, objective, and capable of discrimination and can be combined with other assessment strategies to contribute to a comprehensive student assessment strategy for use in nursing education (4). Therefore, the MCQ examination is not a completely transparent examination tool and is supplemented by short-answer question (SAQ). However, since it is relatively cheap, easy to standardize and rapidly generates an objective score, it is still used to assess (5). Assessment is the process by which the teacher and the student gain knowledge about student progress. Assessment systems should aim at evaluating the desired learning outcomes. It was also demonstrated that a single assessment does not fulfill all aspects of assessment and that there is a need for an evaluating system with multiple ways of assessment (6). In another study, Rassaian showed that the most valid assessment tool was the SAQs. She concluded that it is recommended to use SAQs and true-false questions as the main components of examination, instead of MCQ alone, (7).

Tests that require effortful retrieval of information, such as short-answer tests, promote better retention and greatest gains in memory than tests that require recognition, such as multiple-choice tests (8,9). Therefore, tests can also directly affect learning by promoting better retention of information, a phenomenon known as the testing effect (10). Examination style may enhance study and learning approaches in students (11). Then, studying is different for different types of tests. It is important to learn how to study for multiple choice, true/false, short answer, and essay tests (12).

The purpose of the study is to explore whether the assessment method selection would significantly affect the studying and learning approaches of undergraduate nursing students based on comparing MCQ and SAQ examination scores.

 

 

This study was designed as a descriptive-analytical research. The study has been conducted according to the declaration of Helsinki. One hundred and ninety one first-year undergraduate nursing students enrolled in the study. Students have given the informed consent. The project was conducted in Mother School (Kerman Nursing) and two 

METHODS

pertaining nursing schools (Zarand Nursing and Bam Nursing Schools) and was composed of two separate second-semester student groups. This course has two credits with defined curriculum that have provided 34 hours of didactic lectures. In our study, the course relied primarily on lecture-based teaching with power point presentation and the main form of assessment used in the curriculum was MCQ and SAQ examinations. All of students were given mid and final term examinations with half of MCQs and half of SAQs, consecutively. The composite mid-term examination consists of a 10 multiple-choice questions paper and 10 SAQs paper and final-term examination consists of a 15 MCQs paper and 15 SAQs paper. The students were required to answer two examination papers separately for mid and final-term.

At the end of the final course examination, two structured questionnaires were distributed to ask the students to rank the two different assessment methods based on their preference. Apart from first structured questionnaire that composed of the demographic and academic characteristics of each nursing student, the second questionnaire sought information on student’s studying and learning approaches that may affect the student's learning outcome.

The second structured 12-item questionnaire (MASLAQ) (13) designed based on a modified 5-point Likert scale in outline of visual analogue scale to ask students’ studying and learning approaches. In our study, visual analogue scale including a horizontal line which was divided to two equal parts by a zero point. Choices of each query based on 5-point Likert scale were located on two equal parts of line that joint through a zero point. Each part of line related to an assessment method. On the Likert scale (1 = strongly disagree, 4 = strongly agree and 0= no difference), MCQ examination separate from SAQ examination was marked. To answer to each query of the second 12-item questionnaire, according to students’ personal viewpoint, students were then allowed place a mark on only one of square of related below line on the scale from 0 to 4 for only one exam.

Face validity of 12-item questionnaire was confirmed appropriate. One-week test-retest reliability of the 12-item questionnaire was measured by Cronbach’s alpha at 0.81 and 0.81, respectively. Reliability of the validated 12-item questionnaire was measured by Cronbach’s alpha at 0.77 (13). Range of agreement coefficient of kappa and Pearson’s correlation between pre and post-test questionnaire queries were computed in another article (13).

Statistical analysis was done with the Chi-square test, independent t-test, one way analysis of variance and ANCOVA. SPSS 16.0 was used for statistical analysis and p < 0.05 was considered significant.

 

 

Male students comprised 36.7% of the total participants. Mean (SD) admittance age to university for male and female students was 19.68 ± 3.26 and 19.06 ± 2.14 years, respectively. Baseline characteristics of nursing students according to schools are shown in Table 1.

There was a response rate of 99 percent for completing second questionnaire (189/191). Generally, students favorably

RESULTS

and significantly rated the queries of questionnaire on SAQ assessment part higher than in MCQ assessment part. In order to avoidance of confusing and more precise interpretation of collected data from second questionnaires, a shift is necessary from an assessment method to another method. Then, MCQ assessment’s viewpoints were accurately converted to SAQ assessment’s viewpoints. Subsequently, the scores of strongly disagree, disagree, agree, and strongly agree for multiple-choice option modified to strongly agree, agree, disagree, and strongly disagree short-answer option. Table 2 shows overview frequency of nursing students’ scores for SAQ assessment. With integration of disagree and strongly disagree under one item and integration of agree and strongly agree under another item, analytical analysis allowed us to interpret and deduce agreement between two assessment methods (Table 3).

Female students who participated in the study performed 

significantly better in the learning outcomes in SAQ and MCQ examinations grades and the total grade of both examinations was better than male students who participated in the study (p= 0.001, 0.001 and 0.030, respectively) (Table 4). Table 5 shows responses frequency to studying and learning approaches questionnaire according to lowest and highest quartiles of students’ MCQ and SAQ marks. Most students selected short-answer assessment method with attention to majority queries.

1. Motivation for more profound course study, 2. More time-consuming study for answering to examination, 3. More expanded study for answering to examination, 4. Better and more profound understanding of course meanings, 5. Thinking, deliberation, interpretation and analysis of knowledge in learning process, 6. Capability of retrieval of knowledge in future7. Appropriate evaluation method for nutrition course, 8. Appropriate evaluation method for knowledge level, 9. Appropriate evaluation 

Table 1. Baseline characteristics of nursing students according to schools

Variables

Kerman School

Zarand School

Bam School

P value

No of participations

91 (%47.6)

44 (%23.0)

56 (%29.3)

 

Male sex

34 (%37.4)

14 (%31.8)

22 (%39.3)

 

Admittance age to University

19.6 ± 3.6

18.9 ± 1.1

19.0 ± 1.0

0.244

Place of residency

 

 

 

with Family

38 (%41.8)

4 (%9.1)

7 (%12.5)

 

Dormitory

52 (%57.1)

27 (%61.4)

49 (%87.5)

0.000

Rental house

1 (%1.1)

13 (%29.5)

0 (%0.0)

 

Admission allocation

 

 

 

Aborigine & non-aborigine

79 (%86.8)

44 (%100.0)

53 (%94.6)

0.02

Previous semester GPA

 

 

 

Male

15.27 ± 1.55

15.56 ± 0.92

14.16 ± 1.38

0.005

Female

15.33 ± 1.25

16.37 ± 1.00

15.38 ± 1.40

0.001

Current semester GPA

 

 

 

Male

15.28 ± 1.50

15.87 ± 1.39

14.67 ± 1.70

0.076

Female

16.08 ± 1.07

17.20 ± 1.22

16.65 ± 1.87

0.002

Nutrition examination mark

 

 

 

Male

13.61 ± 3.01

14.59 ± 2.30

12.94 ± 2.18

0.196

Female

14.42 ± 2.45

15.88 ± 1.97

14.64 ± 2.72

0.027

Short-answer assessment mark

 

 

 

Male

6.77 ± 1.71

6.99 ± 1.65

6.15 ± 1.46

0.245

Female

7.33 ± 1.50

7.92 ± 1.15

7.27 ± 1.80

0.161

Multiple-choice assessment mark

 

 

 

Male

6.84 ± 1.51

7.61 ± 0.98

6.80 ± 0.95

0.121

Female

7.09 ± 1.21

7.96 ± 1.02

7.37 ± 1.19

0.005

 

Table 2. Overview frequency of nursing students when multiple-choice questions scores shifted to short-answer questions scores (Modified data)

 

Short Answer Question

Queries

No Difference

Strongly Disagree

Disagree

Agree

Strongly Agree

Motivation for more profound course study

15 (7.9)

42 (22.0)

33 (17.3)

47 (24.6)

54 (28.3)

More time-consuming study for answering to examination

12 (6.3)

11 (5.8)

30 (15.7)

58 (30.4)

80 (41.9)

More expanded study for answering to examination

21 (11.0)

26 (13.6)

43 (22.5)

54 (28.3)

47 (24.6)

Better and more profound understanding of course meanings

13 (6.8)

38 (19.9)

28 (14.7)

46 (24.1)

66 (34.6)

Thinking, deliberation, interpretation and analysis of knowledge in learning process

12 (6.3)

51 (26.7)

28 (14.7)

50 (26.2)

50 (26.2)

Capability of retrieval of knowledge in future

16 (8.4)

32 (16.8)

27 (14.1)

67 (35.1)

47 (24.6)

Appropriate evaluation method for nutrition course

25 (13.1)

42 (22.0)

45 (23.6)

37 (19.4)

40 (20.9)

Appropriate evaluation method for knowledge level

17 (8.9)

32 (16.8)

42 (22.0)

57 (29.8)

41 (21.5)

Appropriate evaluation method for other theoretical courses

28 (14.7)

38 (19.9)

57 (29.8)

37 (19.4)

29 (15.2)

Acquisition of higher grade in examination

15 (7.9)

78 (40.8)

38 (19.9)

26 (13.6)

32 (16.8)

High validity in acquired grade in examination

21 (11.0)

26 (13.6)

23 (12.0)

46 (24.1)

73 (38.2)

High satisfaction with examination

18 (9.4)

73 (38.2)

47 (24.6)

24 (12.6)

27 (14.1)

 

Table 3: Overview frequency of nursing students when multiple-choice questions scores shifted to short-answer questions scores with merging two items (Modified data)

Query

Disagree & Strongly Disagree

Agree &

Strongly Agree

Sig.

1

75 (39.3)

101 (52.9)

0.05

2

41 (21.5)

138 (72.5)

0.000

3

69 (36.1)

101 (52.9)

0.014

4

66 (34.6)

112 (58.6)

0.001

5

79 (41.4)

100 (52.4)

0.117

6

59 (30.9)

114 (59.7)

0.000

7

87 (45.5)

77 (40.3)

0.435

8

74 (38.7)

98 (51.3)

0.067

9

95 (49.7)

66 (34.6)

0.022

10

116 (60.7)

58 (30.4)

0.000

11

49 (25.7)

119 (62.3)

0.000

12

120 (62.8)

51 (26.7)

0.000

 

 

examinations (14). In our study, it is obvious that female students performed significantly better in the learning outcomes, generally. Nonetheless, our first question was whether the change of the assessment type influences students’ learning outcomes and students’ studying and learning approaches and whether we can analyze learning levels.

The findings allowed us to interpret and deduce agreement between two assessment methods (Table 3). With exception of four queries, students mostly agreed with SAQ than MCQ. In a research, the medical students’ perspective regarding three assessment methods revealed that structured short answer assessment was regarded as the preferred modality and the MCQ was the least favored assessment method. Formative assessment is a potentially powerful method to direct learning behavior (15). Southwick et al. indicated that MCQ type’s disadvantages are low expectations for students, encouraging short-term memory and discouraging understanding and long-term memory. These methods also fail to stimulate active participation, collaborative learning, and two-way communication with the professor, and they do not respect the students' diverse talents and ways of learning (11). Although, the MCQ type of tests are commonly used in assessment of knowledge acquisition due to reliability, validity, relatively cheap, easy to standardize, cost-effectively  and rapidly generates an objective score in assessing medical knowledge (3-5). Tests only MCQs are often inadequate to assess knowledge acquisition and may encourage students to memorize abstract textbook knowledge (16). Indeed, when students responding to both SAQs and MCQs, are able to assess the likelihood of answering questions correctly on a moment-by-moment basis, even though they are not able to generate an accurate self-assessment of overall performance on the test (17). An investigation demonstrated that the utilization of examinations containing SAQs created a more

DISCUSSION

 

There was an emphasis on many curriculums on student acquisition of knowledge and this was reinforced by the use of theoretical examinations 

Table 4: Assessment (Examination & Questionnaire) outcomes of nursing students

variables

Male

Female

Sig.

Nutrition examination mark

13.60 ± 2.67

14.84 ± 2.48

0.001

Short answer mark

6.62 ± 1.63

7.46 ± 1.53

0.000

Multiple choice mark

6.98 ± 1.28

7.38 ± 1.20

0.030

P value

0.017

0.486

 

Questionnaire scores

19.6 ± 3.6

19.6 ± 3.6

0.467

Table 5. Responses frequency to studying and learning approaches questionnaire according to quartiles of students’ multiple-choice and short-answer marks

 

Multiple-Choice Test Mark

Short-Answer Test Mark

 

Lowest Quartile

Highest Quartile

Lowest Quartile

Highest Quartile

 

Disagree & Strongly Disagree

Agree &

Strongly Agree

Sig.

Disagree & Strongly Disagree

Agree &

Strongly Agree

Sig.

Disagree & Strongly Disagree

Agree &

Strongly Agree

Sig.

Disagree & Strongly Disagree

Agree &

Strongly Agree

Sig.

1

21

34

0.080

18

24

0.355

22

23

0.881

18

24

0.355

2

13

41

0.000

7

38

0.000

13

31

0.007

10

36

0.000

3

22

27

0.475

12

30

0.005

17

27

0.132

14

29

0.022

4

17

40

0.002

15

28

0.047

18

29

0.109

15

30

0.025

5

24

31

0.345

19

26

0.297

22

24

0.768

18

25

0.286

6

16

34

0.011

16

27

0.093

13

30

0.010

16

27

0.093

7

25

25

1.000

22

18

0.527

24

19

0.446

20

17

0.622

8

20

35

0.043

15

26

0.086

19

26

0.297

14

26

0.058

9

26

22

0.564

23

18

0.435

22

18

0.527

24

14

0.105

10

38

15

0.002

28

14

0.031

34

11

0.001

22

17

0.423

11

17

36

0.009

9

35

0.000

16

27

0.093

11

32

0.001

12

39

14

0.001

32

9

0.000

35

10

0.000

26

15

0.086

 

 

 

challenging learning environment that motivated students to adopt more effective study. The class given examinations with half SAQs along with half MCQs had a significantly higher average score and grade category distribution than the class given examinations with all MCQs or with all SAQs (18). In our research, there is not noteworthy difference between SAQ and MCQ exam mean scores. It is possible; having information about assessment method is an important ingredient of an individual's preparation for the examination. Then, it can be said that a major element of students' preparation depends on their previous conception of the assessment method. This is one of two reasons lack of remarkable difference between SAQ and MCQ exam mean scores. The findings of McKendree and Snowling supported 

 

the current view that a variety of assessment types should be included in the assessment of all medical students, as is already considered to be best practice (2). Rassaian, after analyzing students' score and excluding questions with a discrimination index of less than 0.3, showed that the most valid assessment tool was the short-answer questions (7). Furthermore, an initial short-answer test produced greater gains on a final test than did an initial multiple-choice test (8). Therefore, Combination PowerPoint lectures and notes with MCQs assessment may have encouraged absent long- term retention or the loss of memory of facts taught during year. Essays and SAQs were combined with MCQs to encourage understanding and recall (19). In one study, students did more poorly on the multiple choice 

examination than on the short answer survey (20). In another study, from six commonly used methods in the model, the highest score identifies the most appropriate method. The objective structured clinical examination was preferred, and the essay and short-answer-question examinations were best (21). 

The findings of the present study shows the selection of SAQ assessment method in compare with MCQ assessment method by professor results in significant modification of students’ learning approaches such as motivation for more profound course study, more time-consuming study for answering to examination, more expanded study for answering to examination, better and more profound understanding of course meanings, capability of retrieval of knowledge in future, and high validity in acquired grade in examination. These approaches represent deep learning of students. Short answer question is hypothesized to enhance the development of deep learning. Student disagreement for SAQ assessment method correspond to selecting learning approaches such as appropriate evaluation method for nutrition course, appropriate evaluation method for other theoretical courses, acquisition of higher grade in examination, and high satisfaction with examination. These approaches represent surface learning of students. In our study, assessment preference is defined as imagined choice between assessment alternatives. Higher viewpoint frequencies in MCQ assessment for theoretical courses as one of assessment method are convenience and simplicity.

In a study, authors reported that students have more positive 

attitudes towards multiple choice tests in comparison to free response tests because they think that these tests are easier to prepare for, easier to take, and thus will bring in relatively higher scores (22). Viewpoint of 6.3-14.7% students was “no difference” for queries of questionnaire means that learning level of some students is strategic. It is interesting that in our research, students with strategic or deep learning had the higher grades mean in SAQs than those with surface learning (Data not shown); however, these differences were not significant for any of queries.

 

CONCLUSION 

Our experience suggests that to change the learning approach may be an effective educational tool to help improving learning outcomes of students. Short answer question is hypothesized to enhance the development of deep learning. A single assessment does not fulfill all aspects of assessment and that there is a need for an evaluating system with multiple ways of assessment.

ACKNOWLEDGEMENTS  

We wish to thank the Research in Education Council for donating the grant E/89-169. We would also like to express our gratitude to colleagues in University Education Center; we express our special thanks to all nursing students which made this research possible.

Conflict of Interest: There are no conflicts of interest about this manuscript.

Funding and Support: The grant E/89-169.

 

  1. Benson N, Hulac DM, Kranzler JH. Independent examination of the Wechsler Adult Intelligence Scale-Fourth Edition (WAIS-IV): What does the WAIS-IV measure? Psychol Assess 2004; 22(1): 121-30.
  2. McKendree J, Snowling MJ. Examination results of medical students with dyslexia. Med Educ 2011; 45(2): 176-82.
  3. Abdel-Hameed AA, Al-Faris EA, Alorainy IA, Al-Rukban MO. The criteria and analysis of good multiple choice questions in a health professional setting. Saudi Med J 2005; 26(10): 1505-10.
  4. Brady AM. Assessment of learning with multiple-choice questions. Nurse Educ Pract 2005; 5(4): 238-42.
  5. Hammond EJ, McIndoe AK, Sansome AJ, Spargo PM. Multiple-choice examinations: adopting an evidence-based approach to exam technique. Anaesthesia 1998; 53(11): 1105-8.
  6. Abraham RR, Upadhya S, Torke S. Student perspectives of assessment by TEMM model in physiology. Adv Physiol Educ 2005; 29(2): 94-7.
  7. Rassaian N. A new methodology for comparison of three-test exam techniques in medical students. J Med Educ 2004, 5(1): 3-10.
  8. McDaniel MA, Roediger HL Iii, 
McDermott KB. Generalizing test-enhanced learning from the laboratory to the classroom. Psychol Bull Rev 2007; 14(2): 200-6.

  • Larsen DP, Butler AC, Roediger HL. Test-enhanced learning in medical education. Med Educ 2008; 42(10): 959-66.

  • Butler AC, Roediger HL. Testing improves long-term retention in a simulated classroom setting. Eur J Cogn Psychol 2007; 19(4-5): 514-27.

  • Southwick F, Katona P, Kauffman C, Monroe S, Pirofski LA, del Rio C, et al. Commentary: IDSA guidelines for improving the teaching of preclinical medical microbiology and infectious diseases. Acad Med 2010; 85(1): 19-22.

  • Gloe D. Study habits and test-taking tips. Dermatol Nurs 1999; 11(6): 439-49.

  • Mahmoodi MR. Validation of studying and learning approaches questionnaire to identify students' studying and learning methods. Stride Dev Med Educ 2014; 10(4): 303-12.

  • Noble C, O'Brien M, Coombes I, Shaw PN, Nissen L. Concept mapping to evaluate an undergraduate pharmacy curriculum. Am J Pharm Educ 2011; 75(3): 55.

  • Hill DA, Guinea AI, McCarthy WH. Formative assessment: A student perspective. Med Educ 1994; 28(5): 394-9.

  • Schubert S, Schnabel KP, Winkelmann A . Assessment of spatial anatomical knowledge with a 'three-dimensional multiple choice test (3D-MC). Med Teach 2009; 31(1): e13-7.

  • McConnell MM, Regehr G, Wood TJ, Eva KW. Self-monitoring and its relationship to medical knowledge. Adv Health Sci Educ Theory Pract 2011; 17(3): 311-23.

  • Pinckard RN, McMahan CA, Prihoda TJ, Littlefield JH, Jones AC. Short-answer examinations improve student performance in an oral and maxillofacial pathology course. J Dent Educ 2009; 73(8): 950-61.

  • Southwick FS. Theodore E. Woodward Award: Spare me the PowerPoint and bring back the medical textbook. Trans Am Clin Climatol Assoc 2007; 118: 115-22.

  • Walubo AV, Burch V, Parmar P, Raidoo D, Cassimjee M, Onia R, Ofei F. A model for selecting assessment methods for evaluating medical students in African medical schools. Acad Med 2003; 78(9): 899-906.

  • Lea SF, Anema MG, Brisco VJ, Allie H. The nursing process: What do students know? Abnf J 2001; 12(1): 3-8.

  • Traub RE, McRury KA. Multiple-choice vs. free-response in the testing of scholastic achievement, Ontario Institute for Studies in Education; 1990.