Weekly homework quizzes as formative assessment for Engineering students are a fair and effective strategy to increase learning?

HAL Id: hal-01849941

https://hal.archives-ouvertes.fr/hal-01849941

Submitted on 26 Jul 2018

HAL

is a multi-disciplinary open access archive for the deposit and dissemination of sci- entific research documents, whether they are pub- lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire

destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Engineering students are a fair and effective strategy to increase learning?

Sandra Gaspar Martins

To cite this version:

Sandra Gaspar Martins. Weekly homework quizzes as formative assessment for Engineering students

are a fair and effective strategy to increase learning?. INDRUM 2018, INDRUM Network, University

of Agder, Apr 2018, Kristiansand, Norway. �hal-01849941�

Weekly homework quizzes as formative assessment for Engineering students are a fair and effective strategy to increase learning?

Sandra Gaspar Martins

1

Universidade Nova de Lisboa, Faculdade de Ciências e Tecnologia, Portugal, [email protected]

A strategy to apply online weekly homework quizzes as formative assessment for Engineering students was designed and tested in order to study if it increases student’s learning. The strategy was to make optional weekly online quizzes with questions not randomly generated that students may retry over and over again until to reach the correct answer, they contribute to 10% of grade but only if students get 45% or more in usual pencil and paper assessment.

The quizzes were applied to two different mathematics courses (Single and Multivariable Calculus) of two different Engineering degrees, each one to around 100 students and during a semester. Student’s adherence was very high, nearly all students refer quizzes as fair and useful to learning. Students’ grades were compared with several other years.

Keywords: The role of digital and other resources in university mathematics education, Assessment practices in university mathematics education, Teaching and learning of analysis and calculus.

INTRODUCTION

Frequent online quizzes have been suggested as a strategy to enhance learning by several institutions and researchers. The National Centre for Public Policy and Higher Education in the U.S.A (Twigg, 2005) consider computer based continuous assessment and feedback to be a key strategy for quality improvement in learning.

According to Gibbs (2000), student assessment is an effective way to increase understanding and online quizzes force students to spend more time working productively outside of class. Tuckman (1998) refers this as being especially valuable to procrastinators. One method that can be used to address the crisis in college mathematics, according to Thiel, Peterman, and Brown (2008)

is to ‘provide regular assessment of progress’ and they state that ‘online homework and quizzes with online grading provide students with immediate feedback, the opportunity to correct their homework mistakes, and ongoing assessment of their success in the course’. Booth (2012) considers that homework should be given out at regular times, over regular intervals, on a weekly basis; proposing that learning is work and students should develop regular work habits in order to succeed. Feedback is crucial for student success but giving adequate feedback with large class sizes is difficult and therefore automated systems are a useful solution to the large class size problem.

Quizzes are part of several successful approaches with different kinds of students,

both in top universities and in other higher education institutions. Examples include:

TEAL (Dori & Belcher, 2004) at Massachusetts Institute of Technology (MIT);

SCALE-UP (Beichner, et al, 2007) at North Carolina State University; Peer Teaching (Lasry, Mazur, & Watkins, 2008) at Harvard University.

Particularly, in higher education mathematics teaching, several approaches have been raised but literature is not yet in agreement about the effectiveness of quizzes to enhance learning (Siew, 2003; Varsavsky, 2004; Myers & Myers, 2007; Blanco, Estela, Ginovart & Saà, 2009; Lim, Thiel & Searles, 2012; Broughton, Robinson &

Hernandez-Martinez, 2013; Shorter & Young, 2011).

CONTEXT

This research took place in two mathematics’ courses to Engineering students of Instituto Superior de Engenharia de Lisboa, Portugal, each during a semester. In those semesters, weekly online quizzes on Moodle (the learning management system of the institute) were made available for a week each. The AM2 course in 2013/14 was about Multivariable Calculus, the MAE course in 2015/16 was about Single Variable Calculus. Around 100 students and 3 teachers were involved in each course.

The quizzes were called ‘Mini-tests’ to reinforce their relevance. The ‘regular’

assessment involved two face-to-face tests or the First Exam or the Second Exam.

For AM2, the quizzes scored up to two values proportional to the best 12 (out of 14) grades in the quizzes and it was added if the student scored more than 9.0 values (out of 20) in ‘regular’ assessment. For MAE, it was slightly different: the quizzes valued 10% of the grade if the student scored more than 9.0 values (out of 20) in the

‘regular’ assessment and if this grade was better than the ‘regular’ grade. In both cases the quizzes were optional.

The aim of the quizzes was not to assess students, it was to make them study more, not to postpone, not to study first the other subjects that were naturally more pleasant for them (since they belong to their study area); to make students more aware of their level of understanding (often students only realise that they cannot solve the exercises when they go to the first test, in the middle of the semester). Students are usually optimistic about their capabilities (Wandel, 2015). It was written in Moodle and teachers repeatedly reminded students that the aim of the quizzes was to make students study more and be aware of their level of understanding; that students could copy all quizzes but, probably would not get the 9.0 values required in ‘regular’

assessment and therefore, it not be worthwhile.

THE QUIZZES

The quizzes were produced through the ‘Moodle activity: test’. It allows the

introduction of images and mathematical symbols using LaTeX (see Fig. 1). The

possibility of creating questions with different instances for each student was

considered, but it would take much more time to create questions and students also

know how to solve a problem with a constant instead of a number, so it did not seem

worthwhile.

Figure 1. Multiple-choice questions including a figure and mathematical text, MAE and AM2 example. (Translated)

Whenever it was possible, we used numeric or short answers instead of multiple- choice answers since in multiple-choice answers, with a few tries, students could get the correct answer. The type of questions that we most used was ‘embedded answers’, because this enables a teacher to embed more than one sub-question and those sub-questions may be chosen from all the different question types: numeric, short answers, multiple-choice, true or false, etc. The ‘embedded answer’ question type allows the teacher to evaluate the student through their pathway and not only their final result (see Fig. 3). The feedback does not show the correct answer.

Figure 2. A question with multiple embedded questions along the path (including numerical answers), an AM2 and MAE example. (Translated)

RESEARCH DESIGN, DATA AND RESULTS

This research design is a quasi-experience (not an experience since not all variables

could had been controlled) where two sets of quizzes were applied to two

mathematics courses. The research question of this study is: are the quizzes (applied

with this strategy) a fair and effective tool to increase students’ learning? The strategy

for application of the quizzes is that they are weekly, online, non-mandatory, count

towards grades if students achieve a certain level on traditional assessment, are not

randomly generated and students may resubmit without penalty. This research

question was split into four sub-questions:

• RQ1: Did the students adhere to the quizzes?

• RQ2: What was students’ perception of the quizzes?

• RQ3: Did students felt quizzes as unfair?

• RQ4: Did the quizzes increase students’ grades?

The instruments utilized were: a students’ survey about the quizzes; data from the answers to the quizzes; and course grades over several semesters. The quizzes were applied to two mathematics courses: AM2 with 104 subscribed students and MAE with 108.

The anonymous survey on Moodle was addressed to all students for each edition. The sample of students who answered the survey was reasonable. From the 104 students subscribed to AM2, all subscribed to Moodle, 65 answered the survey. From the 108 students subscribed to MAE, 94 in Moodle, 61 answered the survey. Moreover, by splitting the students by their grade at the first test (the survey was applied before the second test), the number of students answering the survey with a given grade reasonably correlates to the number of students in general who achieved that grade.

Pearson correlation coefficients are ρ = 0.6 and ρ = 0.5 respectively.

Students of the institute do not have precedencies among courses and may be subscribed to a large number of courses, so it is usual that students subscribe to many courses where, in fact, they do not attempt to achieve success. We may verify this, for example, by noticing that from the 108 students subscribed to MAE only 94 were subscribed to Moodle, so the 14 remaining students did not access anything from the course: syllabus, slides, quizzes etc. Since there is no simple and fair way of identifying these students, in this research we always use the subscribed students to make measures. However, it is relevant to have in mind that it includes those ‘ghost students’.

RQ1: DID THE STUDENTS ADHERE TO THE QUIZZES?

AM2 had 104 subscribed students, 79 attempted regular assessments and 76 students

attempted at least one quiz. All but one of the approved students answered at least

one quiz. The final quiz grade was the average of the best 10 out of 14 grades in

quizzes, so it was natural that the last four quizzes had lower attendance (and for this

reason we modified this rule for MAE, where the best 12 grades were chosen).

Chart 1. The number of students that answered AM2 quizzes split by grade.

MAE had 108 subscribed students, 103 completed regular assessment and 93 students attempted at least one quiz. All approved students answered at least one quiz. The final quiz grades were the average of the best 12 out of 14 grades in quizzes, so it is natural that the last two quizzes had a lower attendance (this rule changed from AM2). It is important to note that, for example, in Q5 the number of students with a total grade was lower than in the other quizzes and the number of attempts to solve the quiz was higher than in the others (326). This shows that students were, in fact, trying to reach the correct answers (this test was particularly large and complex).

Chart 2. The number of students that answered MAE quizzes split by grade. The number of attempts to answer the quiz, registered by Moodle, is in parenthesis.

A large portion of students got a very high grade, but this was natural since students may retry without penalty and the questions were equal to all students, so it was expected that students talk to each other and reach the correct answer.

The quizzes were not mandatory and improved the grade if the student got more than 9 out of 20 values in regular assessment, so it could be expected that many students decided not to take it. However, on a regular basis, nearly half of the subscribed students answered the quizzes.

An objective result was, despite of the optional policy, that students strongly adhered to quizzes. The percentage of subscribed students that answered one quiz was 93/108=86% and 76/104=73%. All the quizzes had a high rate of attendance. Among the students that undertook ‘regular’ assessment, almost all took a quiz and a large percentage got high average grades on the quizzes.

RQ2: WHAT WAS STUDENTS’ PERCEPTION OF THE QUIZZES?

Table 1 shows that, according to the survey, none of the students thought that the

quizzes were of no interest and did not care about the quizzes, while a large

percentage believed that the quizzes reminded them to study, showed them the level

that they were reaching and encouraged them to learn new things; some of those

things they thought they understood but in fact they did not.

AM2 MAE

Total 65 100% 61 100%

Quizzes remind me to study the subject every week. 55 85% 50 82%

Quizzes show me there are things I thought I knew but

I didn’t. 48 74% 53 87%

Quizzes help me to have a better perception of the

level I'm reaching. 47 72% 38 62%

I learn new things answering to quizzes. 33 51% 35 57%

Quizzes have no interest. 0 0% 0 0%

I do not care for quizzes, I just copy the results. 0 0% 1 2%

I do not care for quizzes, I not even copy the results. 1 2% 0 0%

Table 1. Students’ answers to ‘Select ALL the statements that you agree with’ in both surveys.

Chart 3. Percentage of students answers to ‘The quizzes were…’ in both surveys.

Summarising, more than 90% of students found quizzes useful (Chart 3); that they study more due to the quizzes. Students agree that quizzes remind them to study, show them that there were things that they thought that had understood but did not, encouraged them to learn new things and gave them a better perception of level that they were reaching.

RQ3: DID STUDENTS FELT QUIZZES AS UNFAIR?

In daily life as a teacher, teachers tell several times that one reason why they do not use online quizzes is because students may be cheating and it may generate unfairness. To avoid that problem, it was strongly emphasised to students that quizzes were much more relevant as formative assessments than summative assessments;

students could resubmit the quiz without penalty to stimulate them to try to answer by

themselves without fear of being penalised; and a clause was included that the

quizzes only count towards grades if students get 9.0 values (out of 20) in regular

assessments, as in Varsavsky (2004). As result, the answers in the survey to the

question ‘Quizzes generate unfairness?’ show that very few students perceive quizzes

as unfair (see Chart 5).

Chart 4. Percentage of students answers to ‘How do you answer to quizzes?’ in both surveys.

Chart 5. Percentage of students answers to ‘Quizzes generate unfairness?’ in both surveys.

When questioned in the survey, no student stated that they had copied the results (see Chart 4), despite it being reinforced in that question that the survey was automatically anonymous.

Therefore, with this approach, the level of unfairness of quizzes is not considered as relevant.

RQ4: DID THE QUIZZES INCREASE STUDENTS’ GRADES?

Since the goal was that all students achieve a total score in all quizzes, is was expected that quiz grades would not correlate to final grades. This did occur and it was verified using the non-parametric Spearman Rho for AM2 (

) and for MAE (

), since data were not normal (Kolmogorov- Smirnov,

).

According to Chart 6, around 70% of students that answered the survey, believe that

quizzes helped them achieve a higher grade.

Chart 6. Percentage of students’ answers to ‘Without quizzes, I’ve scored…’ in both surveys.

The data of Tables 2 and 3, relate to six responsible teachers/approaches and ten different teachers. The syllabus was essentially the same across the semesters but the approaches were naturally different. In the intervention semesters, the responsible teachers were also different. So, the quizzes were not the only different variable in that semester, thus we cannot attribute grade differences directly to the quizzes. For AM2, the pass rate nearly doubled in that semester, the average grade also increased significantly.

AM2 2010/11 2011/12 2012/13 2013/14 2014/15 2015/16

S1 S2 S1 S2 S1 S2 S1 S2 S1 S2 S1 S2

Subscribed students 101 200 128 153 90 123 80 104 56 66 56 108

Approv. students 27 38 31 41 20 23 12 54 10 19 16 33

Average appr. grade 11.7 11.8 12.3 11.7 13.9 12.4 11.5 11.7 11.5

Pass/Subscribed 27% 19% 24% 27% 22% 19% 15% 52% 18% 29% 29% 31%

Professors A+… A+… A+… A+… A+B A+C D+E F+G

+H D+F D +I J+K+

IL

J+K+

I

Table 2. Grades of AM2 students across ten semesters, the letter representing the coordinator teacher is underlined and the experimental semester is shaded.

The MAE course had, in some editions, five or six quizzes in class. It is curious to note that in the year that there were no quizzes, the pass rate was much lower. And the MAE pass grade and the average grade had the highest value in the experimental semester. However, it may have been a coincidence, we do not have enough data to reach any conclusions, it is just a positive indication.

MAE 2011/12-SI 2012/13-SI 2013/14-SI 2014/15-SI 2015/16-SI

Subscribed students 73 109 121 125 108

Pass students 17 30 58 56 61

Average pass grade 12.7 12.2 13.5 12.7 13.5

Pass/Subscribed 23% 28% 48% 45% 56%

Number of quizzes 0 6 in class 5 in class 5 in class 14 online

Professors A A A A+B B+A

Table 3. Grades of MAE students across five semesters, the letter representing the coordinator teacher is underlined and the experimental semester is shaded

Summarising, as expected, quiz grades do not correlate to final grades; around 70%

of respondents to the survey state that due to the quizzes they achieved a better grade.

The pass rate and the average grade increased significantly in the semesters that the quizzes were applied, which is a positive indicator but cannot be directly attributed to quizzes.

CONCLUSIONS

Two sets of 14 weekly quizzes on Moodle were available to all the engineering students on two mathematics courses (Single and Multivariable Calculus). The online quizzes were not mandatory, counted to grading if the student had more than 9 out of 20 values on traditional assessments, were not randomly generated and students could resubmit without penalty. The research question is ‘Are the quizzes (applied with this strategy) a fair and effective tool to increase students’ learning?’

In the answers to the survey, more than 90% of students found quizzes useful; more than 60% stated that studied more due to the quizzes; students agreed that quizzes reminded them to study; showed them that there were things that they thought they understood but did not; made them learn new things and gave them a better perception of the level that they were reaching.

The quizzes were not mandatory so students may have just ignored them. Although a large proportion of students attempted quizzes and kept answering them until the last ones.

Quiz questions were not randomly generated, so all students got the same questions and naturally, students shared the solutions with each other. To avoid unfairness, it was strongly emphasised that quizzes were important to students’ formative assessment, to allow them to test themselves and get feedback on their level of understanding. Moreover, quizzes only contributed to grades if the students got more than 9 out of 20 values in ‘traditional’ assessments. Moreover, if a student copied many quiz results, probably would not achieve the minimum grade and it would not be worthwhile. The result was that, in the answers to the surveys, very few students stated it as being unfair. Over 70% of respondents to the surveys stated that due to the quizzes they achieved a better grade. The pass rate and the average grade increased significantly in the semesters that the quizzes were applied, which is a positive indicator, but it cannot be directly attributed to the quizzes.

This research suggests that these quizzes, with this strategy, are a fair and useful tool to increase students learning.

REFERENCES

Beichner, R., et al. (2007). The SCALE-UP project. Research-based reform of university physics, 1(1), 2-39.

Blanco, M., Estela, M., Ginovart, M., & Saà, J. (2009). Computer assisted assessment through moodle quizzes for calculus in an engineering undergraduate course.

Quaderni di Ricerca in Didattica, 19(2), 78-83.

Booth, D. J. (2012). Managing Mathematics with CALMAT. International Journal of Innovation in Science and Mathematics Education, 2(1).

Broughton, S., Robinson, C., & Hernandez-Martinez, P. (2013). Lecturers’

perspectives on the use of a mathematics-based computer-aided assessment system. Teaching mathematics and its applications, 23, 1-14.

Dori, Y. & Belcher, J. (2004). Improving Students’ Understanding of Electromagnetism through Visualizations-A Large Scale Study. In Paper submitted to the 2004 NARST Annual Meeting, Vancouver.

Gibbs, G. (2000) Changing student learning behaviour outside of class. The Professional & Organizational Development Network in Higher Education. 11.

Lasry, N., Mazur, E., & Watkins, J. (2008). Peer instruction: From Harvard to the two-year college. American Journal of Physics, 76(11), 1066-1069.

Lim, L., Thiel, D. & Searles, D. J. (2012). Fine tuning the teaching methods used for second year university mathematics. International Journal of Mathematical Education in Science and Technology, 43(1), 1-9.

Myers, C., & Myers, S. (2007) Assessing assessment: The effects of two exam formats on course achievement and evaluation. Innovative Higher Educ., 31, 227- 36.

Shorter, N., & Young, C. (2011). Comparing assessment methods as predictors of student learning in an undergrad. mathematics course. IJMEST, 42(8), 1061-67.

Siew, P. F. (2003). Flexible on-line assessment and feedback for teaching linear algebra. I. J. of Mathematical Education in Science and Technology, 34(1), 43-51.

Thiel, T., Peterman, S. & Brown, M. (2008) Addressing the Crisis in College Mathematics: Designing Courses for Student Success, Change: The Magazine of Higher Learning, 40:4, 44-49.

Tuckman, B. W. (1998). Using tests as an incentive to motivate procrastinators to study. The Journal of Experimental Education, 66(2), 141-147.

Twigg, C. A. (2005). Course Redesign Improves Learning and Reduces Cost. Policy Alert. National Center for Public Policy and Higher Education.

Varsavsky, C. (2004). Can online weekly quizzes contribute to learning in mathematics. In Proceed. of 9th Asian Tech. Conf. in Mathematics, Singapore.

Wandel, A. P., Robinson, C., Abdulla, S., Dalby, T., Frederiks, A., & Galligan, L.

(2015). Students’ mathematical preparation: Differences in staff and student

perceptions. I. J. of Innovation in Science and Math. Education, 23(1).