Andreas Janson1, Sissy-Josefina Ernst1, Katja Lehmann1, Jan Marco Leimeister1,2
1 Kassel University, Germany
{andreas.janson, sissy.ernst, katja.lehmann, leimeister}@uni-kassel.de
2 University of St.Gallen, Switzerland JanMarco.Leimeister@unisg.ch
Abstract. Large-scale lectures are a typical way of teaching university students.
However, these lectures often lack interaction elements and do not foster awareness and reflection in the learning process. This results in insufficient learning outcomes such as learning satisfaction and success. Therefore, a new approach to engage interaction in such large-scale lectures is the flipped class- room concept which seeks to overcome these challenges by stimulating self- regulated learning phases and improving interaction as well as awareness and reflection in the presence phases of a lecture. However, it is still unclear how to actually increase reflection and awareness through interaction in such learning scenarios. For this purpose, we propose an application of a technology- enhanced peer assessment that is carried out in large-scale information systems lectures. Preliminary evaluation results suggest the potentials of this approach.
Thus, we are able to provide first theoretical and practical implications for the application of a technology-enhanced peer assessment in large-scale lectures.
Keywords: Awareness, Reflection, Peer Assessment, Large-Scale Lectures, Learning Success, Interaction, Feedback, Educational Objectives
1 Introduction
Large-scale lectures with an uneven lecturer-learner proportion (sometimes more than 100 learners per lecturer) are common in learning scenarios of universities [1]. These lectures are characterized by high anonymity and suffer from a lack of interaction in the learning process - not only among learners themselves but also among learners and lecturers [2]. Often, this results in insufficient learning outcomes and brings about unsatisfied learners [3, 4]. This development is alarming since fundamental elements of learning success include the opportunity to ask comprehension questions in order to get feedback, the possibility of sharing one's opinions concerning the learning con- tent and of intensively reflecting on the learning content with colleagues [5, 6].
35
Moreover, dealing and interacting with the learning content during the learning pro- cess creates awareness and reflection regarding the learning process [7]. Additionally, interaction and collaborative learning with peers are regarded as significant predictors in terms of learning success [8] and positively influence the long-term satisfaction of learners [9, 10]. Individual learning success verification, namely in the teaching- learning process, provides individual feedback to learners [11]. This allows learners and lecturers to identify missing knowledge and misunderstandings not during the final exam, but rather early in the course of a continuous learning-progress monitoring system [11] and moreover, create awareness for the relevant specific learning content.
Integrating assignments in class which create awareness and reflection to the specific learning content are very complex and addresses the high cognitive level of educa- tional objectives [12] supposed by Bloom [13] and Anderson et al. [14], which are analyzing, evaluating and creating. However, the verification of those assignments is time- and resource-consuming hence impossible to use in a large-scale lecture. Never- theless, introducing interaction and feedback to create awareness and reflection and moreover addressing educational objectives on a high cognitive level for individual learning success measurement in a large-scale lecture is a widespread problem.
Didactic mechanisms are needed in order to overcome the above mentioned factors characterizing traditional large-scale lectures. One promising possibility to enhance interaction and feedback and moreover to address high cognitive levels of educational objectives without massively increasing the workload of lecturers is the use of peer assessment as didactic method [15]. By using peer assessment, learners give each other feedback or credit points in terms of a performance during the learning process according to specifically defined criteria. The goal of this paper is to describe the use of peer assessment as interaction supporting component for addressing awareness and reflection in a university large-scale lecture and ultimately for increasing learning success. This paper therefore aims to answer the research question: How is a peer assessment in a large-scale lecture designed to address interaction and to improve the learning scenario? The contribution of this study is according to Gregor [16] a theory of design and action that enables on the one hand practitioners to design learning scenarios with a technology-enhanced peer assessment, and one the other hand de- rives theoretical implications for future research in engineering IT-enabled learning scenarios.
In order to answer the research question, the remainder of this paper is structured as follows. First, we provide a brief overview of related work that is concerned with our peer assessment. We then subsequently propose our application of the technolo- gy-enhanced peer assessment in our learning scenario. Afterwards, we present our first evaluation results and provide implications of our results in the discussion. In section 6, we highlight limitations of our study and provide on this basis guidance for future research, before the paper closes with a brief conclusion.
2 Related Work
For a few years, awareness and reflection have been, in the context of technology- enhanced learning, increasingly important and capture more attention since it has been recognized that both are key factors in helping to provide personal support in user-centric learning environments [17, 18].
The pedagogical approach aims at making learners aware of their learning behavior and at the same time intends to empower learners in creating own personal learning environments with individual learning resources while discovering their own learning patterns.
2.1 Awareness
In this context, awareness plays a central role focusing on cognitive learning activities and especially on non-observable behavior. Learners should familiarize themselves with their own and individual cognitive processes such as goal-setting, self-evaluation or help-seeking, in order to integrate them into self-regulated learning [19]. Students are confronted and made aware of key actions of their own learning behavior with the intention to make them become aware of their cognitive actions. Thereby, e-learning tools with the possibility to personalize the learning process are able to support awareness. Evaluations indicate that learners feel aware especially of their own efforts and less about the effort of their group members and the members of other groups [20].
2.2 Reflection
Reflection is an important key element in the learning activity as it allows implement- ing continuous improvement in order to cope with complex and permanent changing situations [21]. It is a meta-cognitive process which can be individual and also collec- tive [22], and described as the conscious reevaluation of experience for the purpose of guiding future behavior taking into account feeling, ideas and behavior as well [23].
In the context of technology-based learning, active reflection supports the examina- tion of own achievements as well as the work of peers and pushes for a decentraliza- tion process of problem-solving where learners are challenged and confronted with existing knowledge [21] and finally able to create knowledge [24].
Finding out about a learner’s reflection can be supported by several platforms where learners communicate by sharing reports, problems and solutions concerning their work with peers [25]. Additionally, this exchange enables peers to learn from their peers and at the same time to contribute own work. In this way, learners should take more responsibility of their learning activities and efforts.
37
2.3 Peer Assessment
In the context of awareness and reflection in the learning process, prior research has shown that learners who interact with their lecturers and colleagues are more actively involved in the learning process [26, 27] and achieved better learning outcomes [8].
The lecturer can assess the learning progress by means of the answers and provide direct feedback. The learners have the opportunity to contribute their ideas and thoughts, thus, also initiating new thought processes [28, 29].
The use of peer assessment in class is an essential possibility to introduce interac- tion in a large-scale lectures and to provide formatively individual feedback in the learning progress as well as corresponding interventions by means of technical-based observation processes even in groups with a high number of learners [30, 31]. Moreo- ver, the use of peer assessment is a favorable method to give learners extensive open- ended free text assignments hence to address awareness and reflection, even in large- scale lectures with more than 100 students, without massively increasing the lectur- er’s workload. In the case of peer-assessment, learners give each other feedback or credit points in terms of a performance or results during the learning process accord- ing to specifically defined criteria [32]. Peer-assessment turns learners into experts themselves and gives them a deeper understanding of the learning content [33].
The application of peer assessment in university teaching brings about, above all, the following advantages opposed to an evaluation solely done by the lecturer:
1. Logistically: Lecturers can save precious time if learners give each other feedback and evaluate each other’s academic performance [33].
2. Pedagogically: The learners get a deeper understanding of the learning contents by checking and assessing their colleagues’ responses. By reading works of others, one can deepen one’s own knowledge and develop new ideas by evaluating other points of view [13, 33].
3. Metacognitive: Learners will develop awareness for their own strengths and weak- nesses and will be able to compare and evaluate their own performances, at least to a certain extent [34]. In addition, learners train their abilities to think critically [35, 36] as well as how to evaluate and reflect [37].
4. Affectively: Learners perceive qualitative feedback from their peer group as more valuable than a lecturer’s grade [33].
Therefore, the application of peer assessment does not only relieve the lecturer but turns learners into experts themselves. First observations show that evaluations done by the peer group agree with the lecturers’ evaluations of the learners’ academic per- formances [38]. Furthermore, studies show that regular feedback given by the peer group has a positive effect on the learner’s learning process [39]. In their literature overview, van Zundert et al. [40] point out that there are only a few existing case studies concerning an experimental setting of peer assessment and that this circum- stance prevents specific insights on how peer assessment has to be designed. Scien- tific literature brings up terms such as peer assessment, peer grading, peer review, and peer feedback, among others. For this paper, we use the term of peer assessment
meaning that learners of a peer group assess each other’s performances as well as evaluate it according to relevant criteria without giving each other credit points.
3 Theory-motivated Design of a Peer Assessment
For the improvement of our learning scenario, we draw on a theory-motivated design approach [41, 42] for engineering learning services [43]. Therefore, we base our sub- sequent design decisions on the constructs linked to our phenomena of interest.
In particular, we focus on awareness and reflection as ancillary phenomena as well as on learning outcomes as the main phenomena. Awareness and reflection are closely associated with interaction in a learning scenario. Hence, we implemented a peer assessment in our lecture which supports interaction in the learning scenario and in consequence, awareness, reflection, and ultimately learning outcomes. Figure 1 de- picts our theory-motivated design approach.
Fig. 1. Theory-motivated Design Approach
3.1 Concept of a Large-Scale Flipped Classroom
The concept of the presented peer assessment is part of a didactical concept for the flipped classroom, also known as inverted classroom [44] or inverted lecture [45].
This concept is implemented for the first time within an IS lecture at a German uni- versity. By choosing a learner-centered approach, the objectives are to increase the lecture’s quality as well as to convey learner success and satisfaction. The following figure illustrates the flipped classroom concept. We therefore applied the learner- centered concept, which addresses three types of interaction throughout all phases.
Referring to the work of Moore [46], the figure below differentiates between learner- content-interaction, learner-lecturer-interaction, and learner-learner-interaction.
Interaction
Awareness Reflection
Learning Outcome
Interaction supporting component
+
+ +
+
39
Fig. 2. Concept of a Flipped Classroom Learning Cycle Integrating Three Types of Interaction The shown learning cycle has duration of two weeks; it will be repeated 5 times dur- ing one semester. Each cycle compromises four individual phases which are differen- tiated hereinafter: (1) the first phase can be substituted as self- or private study. The learners study small either video- or script-based learning units provided by the lec- turer in a Learning Management System (LMS). (2) During the next phase, every learner prepares a solution for a part of an extensive open-ended free text assignment within an allocated group. Every group needs to bring their solutions on power point slides; these are used as input for the third phase, namely “collaborative clarification”.
(3) This phase is held in presence. The intention of this phase is to discuss the previ- ously submitted solutions, to consider further aspects of the findings and to emphasize its strengths. It constitutes the operational scenario for the application of a peer as- sessment, which is presented after a short explanation of the fourth phase. (4) The learning cycle ends with the phase of “collaborative application” which is dedicated to the tutorials. During the tutorials, all learners elaborate a common solution. In spe- cific, they work on assignments concerning business process management and con- ceptual data modeling.
3.2 The Application of a Peer Assessment in a Flipped Classroom Scenario The peer assessment imbeds itself in the third phase of the flipped classroom learning cycle. Its main goals are the collaborative clarification and consolidation of the pre- pared solutions submitted to the lecturer as well as gaining a deeper understanding of the learning content. Usually, an interactive learner-lecturer discussion is the method of choice. Similar to a traditional lecture, the third phase addresses the interaction type of learner-lecturer-interaction. In order to improve interaction, awareness, and reflection in the presence phases of the lecture, we developed a technology-enhanced peer assessment process addressing additional learner-learner- and learner-content- interaction. Instead of the lecturer presenting several group solutions, the learners
Collaborative Preparation
Collaborative Application Self-Consistent Preparation
Collaborative Clarification Scenario for the application of a peer
assessment LEARNER-LEARNER
Interaction LEARNER-CONTENT
Interaction
LEARNER-LECTURER Interaction
themselves consider strengths and weak points and revise the solutions taking into account the comments made above. By reading and assessing colleagues’ group solu- tions the learners create awareness and reflection regarding their own group solutions.
They get aware concerning their own strengths and weak points and they receive new ideas concerning the learning content. Hence, the peer assessment took place in a synchronous and written form via an online chat using a web based application. After finishing the peer assessment, the lecturer adopts the role of a moderator and supports the learners in the organization of a feedback loop. Figure 3 illustrates the structure of the presented peer assessment.
Fig. 3. The Process of a Peer Assessment
Aware of the restrictions of our lecture room, we applied a web based application, to enable the learner participating to the process of a peer assessment. We therefore used an etherpad as a collaborative online notepad. Etherpad documents are accessi- ble via web browser and support multiuser usage without having to create multiple user accounts [47]. Being real time, capable etherpads enable people to collaborate on ideas, concepts and brainstorming. The selected etherpad has a chat bar on the right sidebar as well as a basic formatting functionality, and allows anonymous or public access. To help the learners during the peer assessment process, all learners are pro- vided with an etherpad compromising the intended assessment structure. Specifying the date and the learning unit, the created text gets manually stored in text files and uploaded to the LMS. Figure 4 shows one out of four generated etherpad documents during our lecture.
Task d) Task c) Task b) Task a)
Etherpad 2
Etherpad 3 Etherpad 1
Etherpad4 LEARNER-LEARNER
Interaction LEARNER-CONTENT
Interaction
LEARNER-LECTURER Interaction
Collaborative Clarification Peer
Assessment
41
Fig. 4. Technology-enhanced Peer Assessment
4 Evaluation
In order to evaluate our technology-enhanced peer assessment, we surveyed the par- ticipants of our lecture. We therefore provided a paper-based pre- and post-test during the lecture. As stated before, we embedded our peer assessment in a flipped classroom IS lecture. Before the presence phase in the lecture hall and the peer assessment, four groups of students worked collaboratively in an online forum on four different as- signments. Each group prepared group individual presentations to a different assign- ment and posted the assignments to the online forum. Before conducting the lecture, we administered the prepared group assignments to the other groups. In the actual lecture, the other groups assessed the elaborated assignments of the other groups.
Afterwards, the lecturer moderated a discussion of the collaborative peer assessment.
To evaluate our procedure, a pre-test was administered before conducting the peer assessment. Afterwards, the peer assessment was conducted as described above and at the end of the lecture, the post-test was administered. In the survey, 35 learners who participated in the peer assessment process answered voluntarily both parts of the survey, which contained questions regarding the experience with the peer assessment.
All items of the survey were adopted from literature and adapted, if necessary, to our research context. The items for measuring the perception of the peer assessment were adopted from Pearce et al. [48] and perceived learning outcomes were adapted from Eom et al. [49].
The pre-test asked questions about the learners’ experiences and expectations re- garding the peer assessment. The results show that only 26 percent of our sample had previously participated in some sort of peer assessment, i.e. paper-based or technolo- gy-enhanced peer assessment. Considering the phases of the peer assessment, the
majority (60 percent) of the learners expected that both writing and receiving feed- back would contribute the most to their learning outcomes. 25.7 percent expected that receiving feedback and 5.7 percent expected writing feedback would contribute the most to their learning outcomes (including 8.6 percent nonresponse). Figure 5 pro- vides details about the further results of the pre-test. Both items were measured on a five point Likert response format (1 = strongly agree; 5 = strongly disagree). Overall, the results show that the learners expected up front that the peer assessment would be useful as a scaffold in the learning process (PA1). Also, the learners expected that their peers were qualified enough to provide valuable feedback (PA2).
Strongly agree Strongly Disagree
1 2 3 4 5
PA1: As a learning tool, I expect the peer assessment will be useful.
Mean 2.38; SD= 0.85 PA2: I think that my peers are well qualified
to provide me with critical feedback on my work:
Mean 2.51; SD= 0.92
N= 35 Fig. 5. Results of the Pre-Test
After conducting the peer assessment, we administered the post-test. First, we meas- ured which part of the peer assessment influenced learning the most. The results show that the learners’ expectations were confirmed, since 45.9 percent reported that learn- ing results most from both writing and receiving feedback. Figure 6 shows further results of the post-test. All items were rated on a five point Likert response format (1
= strongly agree; 5 = strongly disagree) and provide the mean value and standard deviation (SD) of the responses. In addition, a one sample t-test was conducted in order to evaluate whether the mean values for all of the questions are lower than the neutral value. In consequence, the usefulness of the peer assessment is shown imply- ing the rejection of the null hypothesis �0: � ≥3. The results show that several items were rated under the neutral value of 3, indicating, in general, a good fit of the peer assessment. Additionally, the t-test provides evidence that H0 is not supported by several items (at least p<0.05), and can thus be rejected in these cases. The indicator PA3 was rated as good, on average (2.74) showing the overall usefulness of our tech- nology-enhanced peer assessment. PA4 however did not provide significant results.
Therefore, we are not able to provide evidence that the peers involved in the peer assessment were actually suitable to conduct the assessment. In contrast, with a highly significant result, PA5 shows that our participants were able to improve their solu- tions after the assessment. Considering the learning outcomes in terms of learning success, LO1 was found to be significant and LO2 was found to be insignificant.
These results are not really contradicting, since they demonstrate that our participants felt that the peer assessment itself did not actually affect the learning outcomes and
43
that they have learned as much without participating in the assessment process. Fur- ther analysis of item LO3 showed that on average (2.71) the learners noticed an im- provement of the learning experience and quality.
Strongly agree Strongly Disagree
1 2 3 4 5
PA3: As a learning tool, peer assessment was very useful.
Mean 2.74; SD= 0.74; t= 2.05*
PA4: I thought that my peers did a good job in providing me with critical feedback on my work.
Mean 2.80; SD= 0.86; t= 1.36 PA5: I think that I improved my written work
as a result of the assessment that I received or wrote.
Mean 2.40; SD= 0.65; t= 5.45***
LO1: I feel that I learned as much with the peer assessment as I might have without.
Mean 2.57; SD= 1.17; t= 2.16*
LO2: I feel that I learned more with the peer assessment than without it.
Mean 3.11; SD= 1.02; t= 0.61 LO3: The quality of the learning experience
with the peer assessment is better than with- out it.
Mean 2.71; SD= 0.75; t= 2.25*
N= 35 Fig. 6. Results of the Post-Test
5 Discussion
The present study was designed to determine whether an application of a technology- enhanced peer assessment is a suitable instrument to engage awareness and reflection of the learners by improving interaction in our lecture. Our results indicate that the expectations of the learners concerning the usefulness of the peer assessment were confirmed. This is in line with previous research results, which also indicated the usefulness of peer assessments in higher education scenarios [50]. However, we con- firmed the suitability of the peer assessment for a new learning scenario driven by a rich interaction in large-scale lectures. Surprisingly, our results show that the percep- tion of the learners did not reveal any significance for the suitability of their peers for the assessment process. The present findings seem to be consistent with other research
which found that trust in the peer as an assessor is not a significant predictor for learn- ing outcomes [51].
This is also shown by the qualitative insights we gained during our evaluation. One learner states in the evaluation: “The idea is good, but the other students need to col- laborate better.” This opinion also relates to the results of our post-test, which re- vealed that most of the learners were not satisfied with their peers. Therefore, we acknowledge these issues by suggesting the implementation of further components that actually enhance a rich learner-learner-interaction to provide a useful process of awareness and reflection in the learning process. This is also highlighted by the fol- lowing statement of another student: “The procedure is very good and modern, but it depends very much on the fellow students”. Therefore, we also want to highlight the importance of the faithful appropriation of such a learning method. If the learning methods and structures, such as our technology-enhanced peer assessment, are ironi- cally appropriated, learning outcomes may suffer [52–54]. For instance, we noticed that students ironically appropriated the chat function and did not use it for a purpose- ful discussion. As an implication, we would suggest to guide the learning process and provide best-practices how to use the tool faithfully. However, our results also show that the received peer assessment improved the assignments of the learners. We there- fore highlight the importance of the feedback provided by the peers in order to im- prove learning outcomes.
Considering the learning outcomes, we found evidence that the peer assessment has no significant impact on the subjective learning outcomes in our study. However, subjective perceptions have to be judged carefully, especially in the context of learn- ing success [55, 56]. Therefore, we cannot make any definite prediction on how peer assessments actually improve learning outcomes in a flipped classroom scenario.
Interestingly, further analysis of the learning outcomes showed that the learners no- ticed an improvement of the learning experience and quality. This also relates to our results that the peer assessment as a meta-cognitive scaffold is a useful method to improve the learning process and in consequence increase learning outcomes [54, 57].
To sum up, we sought to address interactivity in the learning process as means to improve awareness, reflection, and learning outcomes in our learning scenario. Con- sidering our evaluation results, we can state that the peer assessment is a useful meth- od for structuring presence phases in a flipped classroom scenario. Hence, we high- light as a practical implication the importance of the learning process and the reflec- tion of the learning outcomes by interacting with it. This procedure also creates awareness of the learning progress, enabling learners to actually improve their self- regulated learning activities which are especially important in flipped classroom sce- narios.
6 Limitations and Future Research
This study of a peer assessment in a flipped classroom is still on-going and therefore comes with limitations regarding the evaluation and application. Concerning the eval- uation, we consider the poor response rate in our evaluation. Typically, our lecture is
45
attended by 150-200 students. Since we did not provide any compensation for partici- pating in our survey, the external value might be affected, because maybe only learn- ers well-disposed to such innovative learning concepts participated. In consequence, we consider this limitation and plan to evaluate the peer assessment in an additional and compulsory longitudinal online survey to account for the evolving nature of inter- action with e-learning components in the learning process. In addition, we plan to evaluate the actual effects of the peer assessment by conducting an experiment with a peer assessment treatment group and a control group.
Furthermore future work should investigate peer assessment as instrument for indi- vidual learning success verification during the learning process. Following Bloom’s [13] suggestion, transfer and verification of learning content should be adjusting to various cognitive levels of educational objectives. In large-scale lectures the verifica- tion of high cognitive levels of educational objectives is very time- and resource- consuming and hence impractical in use. Peer assessment should be investigated as time- and resource-saving manner to measure learning success during the learning process.
The other part of our limitations deals with the on-going application of our peer as- sessment. We applied the peer assessment in our lecture for the first time. Hence, we are still adjusting and modifying the process for the deployment of the peer assess- ment. In consequence, our evaluation could be biased by effects that are induced through the first time application, e.g., glitches that are mainly concerned with usabil- ity issues. However, we seek to overcome these limitations with a broad application during the next terms and further insights by this application. This would also include the application within other learning scenarios, especially those that are influenced by cultural differences [58, 59].
7 Conclusion
This paper has examined the application of a peer assessment in a large-scale IS lec- ture arranged in a flipped classroom setting. We therefore provided first evidence of the utility of peer assessments as suitable instruments to increase awareness and re- flection as well as to strengthen learning outcomes in an IS lecture. The results of this investigation show that the peer assessment itself does not affect the learning out- comes, but it does have a positive impact on learning experience and quality. Alt- hough the current study is based on a small sample of participants, the findings sug- gest that the application of a peer assessment might be a useful instrument to effect awareness and reflection. Considerably more work will need to be done to determine the effects of a technology-enhanced peer assessment on awareness and reflection as well as on learning outcomes.
Acknowledgements
The research presented in this paper was funded by the German Federal Ministry of Education and Research in the project kuLtig (www.projekt-kultig.de), FKZ 01BEX05A13.
References
1. Leidenfrost, B., Strassnig, B., Schabmann, A., Carbon, C.C.: Verbesserung der Studiensi- tuation für StudienanfängerInnen durch Cascaded Blended Mentoring. Psychologische Rundschau 60, 99–106 (2009)
2. Grießhaber, W.: Neue Medien in der Lehre, http://spzwww.uni- muenster.de/griesha/pub/tnmlehre.pdf
3. Lehmann, K., Söllner, M.: Theory-driven design of a mobile-learning application to sup- port different interaction types in large-scale lectures. ECIS 2014 Proceedings (2014) 4. Lehmann, K., Thillainathan, N., Bitzer, P., Leimeister, J.M.: Performance Dashboard für
Dozenten in der universitären Lehre. In: Multikonferenz der Wirtschaftsinformatik (MKWI). Braunschweig, Deutschland (2012)
5. Picciano, A.G.: Beyond student perceptions. Issues of interaction, presence, and perfor- mance in an online course. Journal of Asynchronous learning networks 6, 21–40 (2002) 6. van den Boom, Gerard, Paas, F., van Merriënboer, Jeroen J.G.: Effects of elicited reflec- tions combined with tutor or peer feedback on self-regulated learning and learning out- comes. Learning and Instruction 17, 532–548 (2007)
7. Boud, D., Walker, D.: Promoting reflection in professional courses: The challenge of con- text. Studies in Higher Education 23, 191–206 (1998)
8. Moore, A., Masterson, J.T., Christophel, D.M., Shea, K.A.: College Teacher Immediacy and Student Ratings of Instruction. Communication Education 45, 29–39 (1996) 9. Hardless, C., Nilsson, M., Nuldén, U.: ‘Copernicus’ experiencing a failing project for
reflection and learning. Management Learning 36, 181–217 (2005)
10. Alonso, F., Manrique, D., Viñes, J.M.: A moderate constructivist e-learning instructional model evaluated on computer specialists. Computers & Education 53, 57–65 (2009) 11. Mayer, H.O., Hertnagel, J., Weber, H.: Lernzielüberprüfung im eLearning. Oldenbourg, R
(2009)
12. Atkins, S., Murphy, K.: Reflection: a review of the literature. J Adv Nurs 18, 1188–1192 (1993)
13. Bloom, B.S.: Taxonomy of educational objectives. David McKay New York (1956) 14. Anderson, L.W., Krathwohl, D.R., Airasian, P.W., Cruikshank, K.A., Mayer, R.E., Pin-
trich, P.R., Raths, J., Wittrock, M.C.: A taxonomy for learning, teaching, and assessing: A revision of Bloom's taxonomy of educational objectives, abridged edition. White Plains, NY: Longman (2001)
15. Strijbos, J.-W., Ochoa, T.A., Sluijsmans, D.M., Segers, M.S., Tillema, H.H.: Fostering Interactivity through Formative Peer Assessment in (Web-Based) Collaborative Learning Environments. In: Mourlas, C., Tsianos, N., Germanakos, P. (eds.) Cognitive and Emo- tional Processes in Web-Based Education, pp. 375–395. IGI Global (2009)
47
16. Gregor, S.: The nature of theory in information systems. MIS Quarterly 30, 611–642 (2006)
17. Nussbaumer, A., Scheffel, M., Niemann, K., Kravcik, M., Albert, D.: Detecting and Re- flecting Learning Activities in Personal Learning Environments. Proc. of the 2nd Work- shop on Awareness and Reflection in Technology-Enhanced Learning (#artel12) held in conjunction with EC-TEL 2012, Saarbrucken (Germany) (2012)
18. Bodemer, D., Dehler, J.: Group awareness in CSCL environments. Computers in Human Behavior 27, 1043–1045 (2011)
19. Dabbagh, N., Kitsantas, A.: Supporting Self-Regulation in Student-Centered Web-Based Learning Environments. International Journal on E-Learning 3, 40–47 (2004)
20. Phielix, C., Prins, F.J., Kirschner, P.A.: Awareness of group performance in a CSCL- environment: Effects of peer feedback and reflection. Computers in Human Behavior 26, 151–161 (2010)
21. Cressey, P., Boud, D., Docherty, P.: Book Review: Productive Reflection at Work. Man- agement Learning 37, 410–413 (2006)
22. Høyrup, S., Elkjær, B.: Reflection - beyond the individual. In: Boud, D., Cressey, P., Do- cherty, P. (eds.) Productive reflection at work: Learning for changing organizations, pp.
29–42. Routledge (2006)
23. Boud, D., Keogh, R., Walker, D.: Reflection, turning experience into learning. Kogan Page; Nichols Pub., London, New York (1985)
24. Nonaka, I.: A Dynamic Theory of Organizational Knowledge Creation. Organization Sci- ence 5, 14–37 (1994)
25. Fischer, G.: Understanding, fostering, and supporting cultures of participation. interactions 18, 42 (2011)
26. Liu, T.-C., Liang, J.-K., Wang, H.-Y., Chan, T.-W., Wei, L.-H.: Embedding educlick in classroom to enhance interaction. In:, pp. 117–125 (2003)
27. Wang, M.C., Haertel, G.D., Walberg, H.J.: What influences learning? A content analysis of review literature. The Journal of Educational Research 84, 30–43 (1990)
28. Gagné, E.D., Yekovich, C.W., Yekovich, F.R.: The cognitive psychology of school learn- ing. HarperCollins College, New York (1993)
29. Morgan, N., Saxton, J.: Teaching, questioning, and learning. Routledge, New York (1991) 30. Bischof, L.: Die digitale (R) evolution? Chancen und Risiken der Digitalisierung akademi-
scher Lehre (2013)
31. Piech, C., Huang, J., Chen, Z., Do, C., Ng, A., Koller, D.: Tuned models of peer assess- ment in MOOCs. arXiv preprint arXiv:1307.2579 (2013)
32. Boud, D., Falchikov, N.: Rethinking assessment in higher education. Learning for the longer term. Routledge (2007)
33. Sadler, P.M., Good, E.: The impact of self-and peer-grading on student learning. Educa- tional assessment 11, 1–31 (2006)
34. Darling-Hammond, L., Ancess, J., Falk, B.: Authentic assessment in action. Studies of schools and students at work. Teachers College Press (1995)
35. Block, J.H., Airasian, P.W., Bloom, B.S., Carroll, J.B.: Mastery learning. Theory and practice. Holt, Rinehart and Winston New York (1971)
36. Zoller, U.: Are lecture and learning compatible? Maybe for LOCS: Unlikely for HOCS.
Journal of Chemical Education 70, 195 (1993)
37. Sitthiworachart, J., Joy, M.: Web-based peer assessment in learning computer program- ming. In:, p. 180 (2003)
38. Kulkarni, C., Wei, K.P., Le, H., Chia, D., Papadopoulos, K., Cheng, J., Koller, D., Klem- mer, S.R.: Peer and self assessment in massive online classes. ACM Trans. Comput.-Hum.
Interact. 20, 1–31 (2013)
39. Dochy, F., Segers, M., Sluijsmans, D.: The use of self-, peer and co-assessment in higher education. A review. Studies in Higher Education 24, 331–350 (1999)
40. van Zundert, M., Sluijsmans, D., van Merriënboer, J.: Effective peer assessment processes:
Research findings and future directions. Learning and Instruction 20, 270–279 (2010) 41. Briggs, R.O.: On theory-driven design and deployment of collaboration systems. Interna-
tional Journal of Human-Computer Studies 64, 573–582 (2006)
42. Gehlert, A., Schermann, M., Pohl, K., Krcmar, H.: Towards a Research Method for Theo- rydriven Design Research. Wirtschaftinformatik Proceedings 2009 (2009)
43. Leimeister, J.M.: Dienstleistungsengineering und -management. Springer Berlin Heidel- berg, Berlin, Heidelberg (2012)
44. Strayer, J.F.: How learning in an inverted classroom influences cooperation, innovation and task orientation. Learning Environ Res 15, 171–193 (2012)
45. Gehringer, E.F., Peddycord, B.W.: The inverted-lecture model. In: Camp, T., Tymann, P., Dougherty, J.D., Nagel, K. (eds.) Proceeding of the 44th ACM technical symposium, p.
489 (2013)
46. Moore, M.G.: Editorial: Three types of interaction. American Journal of Distance Educa- tion 3, 1–7 (1989)
47. Karlhuber, S., Wageneder, G., Freisleben-Teutscher, C.F.: Einsatz kollaborativer Werk- zeuge-Lernen und Lehren mit webbasierten Anwendungen. Lehrbuch für Lernen und Leh- ren mit Technologien (2013)
48. Pearce, J., Mulder, R., Baik, C.: Involving students in peer review. Case studies and practi- cal strategies for university teaching. (2009)
49. Eom, S.B., Wen, H.J., Ashill, N.: The Determinants of Students' Perceived Learning Out- comes and Satisfaction in University Online Education: An Empirical Investigation*. De- cision Sci J Innovative Educ 4, 215–235 (2006)
50. Falchikov, N., Goldfinch, J.: Student Peer Assessment in Higher Education: A Meta- Analysis Comparing Peer and Teacher Marks. Review of Educational Research 70, 287–
322 (2000)
51. van Gennip, Nanine A.E., Segers, M.S., Tillema, H.H.: Peer assessment as a collaborative learning activity: The role of interpersonal variables and conceptions. Learning and In- struction 20, 280–290 (2010)
52. Gupta, S., Bostrom, R.: Technology-Mediated Learning: A Comprehensive Theoretical Model. Journal of the Association for Information Systems 10, 686–714 (2009) 53. Gupta, S., Bostrom, R.: An Investigation of the Appropriation of Technology-Mediated
Training Methods Incorporating Enactive and Collaborative Learning. Information Sys- tems Research 24, 454–469 (2013)
54. Gupta, S., Bostrom, R.P., Huber, M.: End-user Training Methods: What We Know, Need to Know. SIGMIS Database 41, 9–39 (2010)
49
55. Sitzmann, T., Ely, K., Brown, K.G., Bauer, K.N.: Self-Assessment of Knowledge: A Cog- nitive Learning or Affective Measure? Academy of Management Learning & Education 9, 169–191 (2010)
56. Benbunan-Fich, R.: Is Self-Reported Learning a Proxy Metric for Learning? Perspectives From the Information Systems Literature. Academy of Management Learning & Education 9, 321–328 (2010)
57. Bitzer, P., Janson, A.: Towards a Holistic Understanding of Technology-Mediated Learn- ing Services - a State-of-the-Art Analysis. ECIS 2014 Proceedings (2014)
58. Janson, A., Peters, C., Leimeister, J.M.: Der Weg zur effizienten Bereitstellung kultursen- sitiver Dienstleistungen - erste Schritte mittels systematischer Modularisierung. In:
Thomas, O., Nüttgens, M. (eds.) Dienstleistungsmodellierung 2014, pp. 266–286 (2014) 59. Hammer, N., Janson, A., Leimeister, J.: Does culture matter? A qualitative and compara-
tive study on eLearning in Germany and China. BLED 2014 Proceedings (2014)
