When evaluating an eLearning project it is necessary to distinguish the evaluation of the entire project from the evaluation of the individual subjects (courses). It is also essential to distinguish evaluation from the submitters’ point of view and those done by participants of the project: learners, teachers, personnel concerned, etc.
For the evaluation of the entire project we analyze data acquired from evaluation materials, e.g. study support evaluation reviews, but also the quality of “secondary products”, e.g. the structure of the study support evaluation questionnaire. In the final evaluation the view at the whole eLearning project is also considerable. Then we can simulate different situations, e.g. what would happen, if we take out one of the subjects?
ACTIVITY OUTPUTS
We have chosen the following evaluation activity outputs: questionnaire, review, checklist, evaluation report, survey record, evaluation statement, data analysis conclusions. To make the process of evaluation less demanding and consuming, we need to optimize the set of evaluation materials. It is possible to cumulate some of the outputs, e.g. in the above mentioned table we can merge all of the evaluation statements of the readiness of the institution to a single checklist.
Which output documents were proposed and filled in with data (Šarmanová in Kapounová, 2012) 1. Basic identification of project – project heading.
2. Project analysis
Check of completeness of eLearning project analysis: a level of need and project contribution, its objectives, target group, plan of key activities, time schedule, quantification of outputs, sustainability
Key project activities: activity title, its description, implementation, output, and activity expenditures.
Quantification of project outputs in individual subjects: subject supervisor, subject title, number of text pages and costs per page, number of multimedia objects and costs of their acquisition, opponents.
Project time schedule (e.g. Gantt diagram): training of authors and tutors, purchase of technology, author’s work, creation of multimedia…pilot courses, evaluation.
Note: Costs of funding of all-project needs result from activities and time schedule.
Project budget; we list here for example units, quantity, unit price, etc. in the structure:
personnel costs, other costs.
3. Formalities – check of analysis completeness: technical readiness, human resources, organization readiness, client approach, readiness to project risks.
4. Project suitability – general project plan: its need and benefit, adequacy of target groups, contribution of didactics, experience of solver, project sustainability, adequacy of budget (related to objectives), project publicity and many others (Milková and Slabý, 2006).
Evaluation of analysis means assessment of the project need and readiness.
If the project is approved for solving, next phase is design. The analysis results are used to control the entire project and to manage each subject as well.
5. Project management – check of general and organizational activities such as training of authors and tutors, contracts with solvers, etc.
94
6. Control of each subject can be realized in the form of the checklist of completeness; again at the level of subject activities (e.g. content of text), followed by organizational activities of the whole project.
Next step is development of subject supports. That is the longest phase of solution and it is evaluated in terms of its result.
7. Development of learning supports and their verbal and score evaluation
lay reader review
expert opinion
pedagogical opinion; detailed analysis (Šarmanová and Kapounová, 2010) modified the original proposal (Mechlová, 2008).
After termination of the development phase, the course is ready for implementation and testing in pilot run either in LMS environment or on the Internet or all study materials can be delivered on CD-ROMs.
8. Assessment of students
The content of tests is defined by subject supervisor. Tests may involve pre-tests, intermediated tests, post-tests, etc. The advantage is automatic evaluation of results. All information can be stored and successively used for various analyses – about the subject, the overall project, individual student, etc.
9. Evaluation of support by student statement, e.g. using evaluation questionnaire (electronic or hard copy).
SUMMARY
Following completion of all project activities we have comprehensive data accumulated about each subject and we have overall and partial financial project costs.
What we discover from the final project evaluation, e.g.
comparison of pilot course results with previous years results;
benefits for students;
benefits for pedagogues.
Then we can deduce about the effectiveness of each subject or of the entire project.
For individual subjects actual data it can be evaluated – mainly on costs (of subject, person involved, pilot student…), results of students’assessment, acquired knowledge and skills, student feedback, use of acquired knowledge and skills in practice (if feasible) and others.
For the whole project it can be assessed – overall costs, but also costs related to individual items (subject, person, etc.), analysis of reviews, questionnaires, and tests and finally the long term benefit.
The described model for the eLearning project evaluation was applied in several steps.
Firstly, expert and pedagogical evaluation was made of 56 study supports developed at three technical faculties (Šarmanová, 2009). The process and results of data analysis from questionnaires was presented at CBLIS conference 2010 (Šarmanová and Kapounová, 2010).
Next step involved evaluation of eLearning project for 27 subjects taught at two universities in Ostrava.
Data was recorded in 14 tables for each subject and then, data on the whole project followed. The structure of evaluation forms was described in this paper. Individual items in questionnaires can be answered (according to the question type) verbally, by points (score), with semantic differential, or just to complete it as checklist.
Conclusions from this analysis were made and its feedback can help in project update. Proposed forms for evaluation were revised and made more precise, so that both project solvers and reviewers understand them clearly.
95
As a part of activities of the project life cycle is running in electronic environment, many data can be recorded automatically and stored data can be subsequently processed with suitable software (e.g. with statistic programs).
Existing eLearning environment enables more than to deliver study material to all learners in a uniform way. It is possible to prepare such course trajectory which could be tailor-made to study characteristics of learners. It is essential to prepare such study supports which satisfy requirements of individual student. The level of success in the development of the adaptive eLearning material can be assessed by the tool for evaluation of eLearning study material in light of the adaptive learning methodology. The adaptive study materials were prepared for several subjects. The attributes which can be considered for the adaptive method of education are evaluated in the specially designed questionnaire (Kapounová and Kostolányová, 2011).
The acquired data can be used for many kinds of analysis including statistic investigation while using e.g. methods for data mining (Šarmanová, 2007). The obtained results can be used as feedback in the whole process of evaluation. Hence, next run of endless cycle of learning support improvement and thus teaching enhancement is initiated.
In any project, it is essential to have a verified systematic process for development and evaluation. Our project utilizes system analysis to define relevant stages of the project. If we understand the eLearning project as development of a group of eLearning study subjects, it will include two levels: the first level is the specific content and the second level is for the entire project. In addition, we define suitable evaluation methods for each eLearning subject. Overall, it brings a detailed plan for eLearning evaluation.
REFERENCES
Åström, E. (2009). E-learning quality. Swedish National Agency for Higher Education, Stockholm, Sweden.
Clark, R. C. and Mayer, R. E. (2002). e-Learning and the Science of Instruction: Proven Guidelines for Consumers and Designers of Multimedia Learning. San Francisco: Pfeiffer.
Huges, G. and Attwell, J. (2003). A framework for the evaluation of e-learning [online], London:
KnowNet, <http://www.theknownet.com/ict_smes_seminars/papers/Hughes_Attwell.html>.
Kapounová, J. (2007). Approaches to the evaluation of eLearning. Proceedings of the Computer Based Learning in Science conference, pp. 208–214.
Kapounová, J. (2012). Přístupy k evaluaci eLearningu. Ostrava: University of Ostrava.
Kapounová, J. and Kostolányová, K. (2011). Specific tool for evaluation of eLearning. Proceedings of Interactive Collaborative Learning. pp. 423-427.
Kapounová, J. and Šarmanová, J. (2010). The Evaluation of eLearning Study Supports. Proceedings of the Computer Based Learning in Science conference, pp. 103–109.
Khan, B. (2005). Managing e-learning: design, delivery, implementation and evaluation. Information Science Publishing.
Mechlová, E. (2008). Tvorba e-learningových kurzů pro technické obory. Ostrava: Technical University of Ostrava.
96
Milková, E. and Slabý. A.(2006). E-learning as a Powerful Support of Education at Universities.
Proceedings of the 28th International Conferences on Information Technology Interfaces. pp. 83–88.
Šarmanová J. (2007). Datové sklady a dolování znalostí z nich. Ostrava: Technical University of Ostrava.
Šarmanová, J. (2009). E-learningové prvky pro podporu výuky odborných a technických předmětů, Ostrava: Technical University of Ostrava.
Systems Development Life Cycle (2011) [online],
http://en.wikipedia.org/wiki/Systems_Development_Life_Cycle.
Winfrey, E. C. (1999). Kirkpatrick's Four Levels of Evaluation. In Hoffman, B. (Ed.), Encyclopedia of Educational Technology. [online], http://coe.sdsu.edu/eet/Articles/k4levels/start.htm.
ACKNOWLEDGMENT
This work was supported by the Czech Science Foundation under the contract No. 406090242.
Jana Kapounová University of Ostrava Dvorakova 7
70103 Ostrava Czech Republic
email: jana.kapounova@osu.cz