The “Secure Exam Environment”: E-Testing with Students’ Own Devices

HAL Id: hal-01762856

https://hal.inria.fr/hal-01762856

HAL is a multi-disciplinary open access

archive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

The “Secure Exam Environment”: E-Testing with

Students’ Own Devices

Gabriele Frankl, Peter Schartner, Dietmar Jost

To cite this version:

The “Secure Exam Environment”: E-Testing with

Students’ own Devices

Gabriele Frankl, Peter Schartner, Dietmar Jost

Alpen-Adria-Universität Klagenfurt, Austria

Gabriele.frankl@auu.at

Abstract. In the 21st century, written exams continue to be the primary method of assessing factual knowledge. Conducting these exams online reduces the correction workload and offers advantages such as enhanced objectivity, assessment with the possibility to use software specific to the course and thus increased constructive alignment with teaching and learning processes. However, eExams are often conducted in spaces that are too small, since larger computer rooms are usually unavailable or not economically feasible. Hence, in June 2011, we implemented a system called Secure Exam Environment (SEE) that enables online testing in any lecture hall with electricity and LAN sockets using students’ own devices while blocking access to unauthorized files or internet pages. Loan devices are offered to students that have no suitable device for the SEE. Assessment is conducted via Moodle and additional software (e.g. Eclipse, GeoGebra) can be used as well. The SEE also addresses important issues such as security, reliability, high availability, privacy, and flexibility. As of August 2017 we have conducted 1,241 such online exams with 46,342 students and are able to test up to 220 students concurrently. Furthermore, we offer students the possibility to choose their preferred time slot to sit an eExam within predefined weeks.

Keywords: Secure online testing, secure exam environment, security, reliability,

high availability, privacy, monitoring.

1. Introduction

Grading is still an important and non-trivial part of modern university life, since it heavily influences student behaviour [1]. The assessment process “can be used as a means of channelling students’ energies, and the feedback that it generates can provide students with an opportunity for reflection” [2]. Various approaches to assessment offer students the possibility to demonstrate their competences using a full range of examination modalities for different types of knowledge.

Next to the noticeable time and money savings [4] due to automatic delivery, storage and the (semi-)automated correction of (semi-)standardized question types, along with the improved readability, structure and clarity of typed open-text answers, eExams offer various advantages and improvements for assessment. Online exams increase student engagement due to their relative novelty and provide greater flexibility if compared to traditional testing methods [4]. The greater efficiency of online exams provides students with instant grading and feedback [5]. Moreover, today's students are not used to extensive handwriting anymore [6], leading to hand pain and bad handwriting when taking paper-and-pencil exams. Furthermore, with eExams different handwriting styles do not influence the lecturer when grading. It is also easier, for example, to evaluate all the answers for a single question at once by switching between the different students’ answers, thus enabling each question to be evaluated on its merits without being influenced by other answers provided by the student. Hence, online exams enhance objectivity in that they are not steeped in subjective construction processes. Additionally, eExams can be saved and shared between lecturers easily, offering the opportunity for (mobile) synchronous correction. Although the creation of online questions might require greater initial effort, the establishment of a question pool leads to a reduction in effort over time, thus guaranteeing the sustainability of online testing. Moreover, the digital format offers opportunities for statistical analysis of questions, improving the quality of questions over time. Finally, the shuffling of questions and answers decreases chances for cheating. Another and very promising advantage of online exams is the opportunity to include additional software and multimedia into the examination environment. Biggs and Tang [7] postulate that a well-founded lecture design includes assessment. Their concept of “constructive alignment” emphasizes the necessity of establishing coherence between all phases and elements of the learning process. Intended learning outcomes, teaching/learning activities, assessment tasks as well as grading should support one another [7][8]. Although Biggs and Tang’s [7] approach has received criticism, we are of the opinion that it is particularly useful in highlighting the power of assessment to shape students’ experiences [9]. Thus, in order to ensure coherence, the software tools used for teaching and learning should be part of the examination process. This means that mathematical calculations and analysis, spreadsheets, literature essays etc., can be easily processed online using appropriate software. Being able to use specific software and multimedia in electronic exam environments facilitates the way to promising (hands-on) performance assessments too.

Despite all the positive aspects of eExams mentioned so far, we found a lack of technical solutions to conduct secure online exams for larger audiences. The problems we encountered were too small computer rooms as well as a lack of consideration of the security requirements which inevitably arise in the context of (electronic) exams: confidentiality, privacy, integrity, authenticity, and availability. The first four aspects are commonly addressed through cryptography (e.g. encryption of transmitted and stored data, and authentication of messages and users), whereas the last one is provided by redundancy and continuous monitoring of the IT system.

2. The Secure Exam Environment (SEE)

The Alpen-Adria-Universität Klagenfurt (AAU) launched the Secure Exam Environment (SEE) for online testing in 2011 [10] with the aim of supporting large class sizes and modern teaching as well as testing strategies, while working within budgetary and organisational constraints. In particular, the SEE makes use of the students’ existing technical resources, specifically their personal computers (laptops). The efficiency of allowing students to use their own devices is complemented by an effectiveness factor since they are presumably familiar with the hardware.

The SEE disables access to students’ own files and data as well as to other internet sites. Loan devices are offered for those who do not own a laptop. As a result, institutional asset requirements as well as the associated maintenance costs are minimized. We are currently able to test up to 220 students simultaneously.

Furthermore, courses are becoming increasingly based on or supported by different software tools and programmes, for example a statistics programme or special mathematical software packages. Traditional testing methods do not allow testing related to the use and application of such software programmes; the SEE, on the other hand, offers this possibility which is consistent with pedagogical coherence [8].

The actual exams are presented as quizzes, a key component of the Moodle learning management system (LMS) utilised by the AAU.

2.1 How do we secure the SEE?

As with traditional exams, cheating is a major problem. On the one hand, the issue of impersonation must be addressed, on the other hand, access to (electronic) materials such as notes, books, (external) devices, or local or online resources which are not allowed during an exam must be prevented.

We discourage impersonation by requiring students to present a valid student ID card. In addition, we utilize the photo stored in each student's Moodle profile to verify that they are indeed using the correct login credentials. During the exam we compare the registration data with the (number of) students who started the exam in the Moodle course. Starting the exam from outside a lecture hall is not possible due to automated grouping of registered students in Moodle after registration, since solely the members of this grouping are able to open the exam (students leave this grouping by checking out with their student card after completing the exam). In addition, only devices with an IP address within the IP-range of the specific lecture hall are allowed to open the exam and the settings of the exam in Moodle only allow access via the Safe Exam Browser: students are only permitted to attempt the exam once, and the exam is available in the Moodle course only at the scheduled time and, once started, only for a certain period.

install additional software (such as lockdown modules) on their personal laptops, we have to use our own operating system (OS) in order to restrict the access to the local resources and programmes that are prohibited during the exam. We decided to boot this OS via the Preboot eXecution Environment (PXE) protocol over a local area network (LAN), since the handling of USB sticks or DVDs is very error-prone, time-consuming and inflexible, especially when additional software is needed, and the usage of WLANs is too insecure and interference-prone. Clearly, this requires that the client is able to boot via the network.

In order to support a very broad range of (private) laptops and prevent students from accessing local resources and programmes, our solution is designed as a minimal Linux system. At the moment, this OS is realized using Fedora and Knoppix, which enables us to boot Legacy or UEFI devices (both Apple and PC). In order to restrict the access to external resources, we implemented corresponding firewall rules. Since Moodle as an LMS not only provides exam features but also chatting capabilities and course related material, a solution was needed to prevent the access to such resources and activities during exams. Running an ordinary web browser in centOS – even when restricted with firewall rules – would not have completely solved the cheating problem. Fortunately, the Safe Exam Browser (SEB – [12]) is fully supported by Moodle-core. The SEB is much more secure than an ordinary browser, since it prevents students from opening other programmes or additional web browser windows during the exam and ignores certain key combinations or clicks. So by restricting the access to the exam page only, cheating by exploiting Moodle’s features is not possible anymore. However, the SEB is only available for Windows XP, Windows 7 and MAC OS X. Therefore, we are forced to boot a minimized Windows 7 as a virtual machine on the minimized Linux system via VirtualBox ([13]) (see Figure 1). Additionally, proprietary software which only runs on Windows systems is still widespread in the educational sector. On the one hand, the reliance on Windows 7 is a drawback in terms of performance, on the other hand, it adds flexibility regarding the management of the virtual machine image. Furthermore, hardware driver management is done completely in Linux, which is known for its broad hardware support especially for older hardware without the need to install specific drivers. The selection of the allowed programmes (beside the SEB) during the exam is set via a configuration file, which is retrieved from an Intranet Service. In the GUI of this service, administrators are able to configure the exam (e.g. only Calculator or Calculator and Excel or GeoGebra or Eclipse and PDFs allowed).

Fig. 1. The operating principle of the Secure Exam Environment (SEE)

2.2 How do we ensure the SEE is reliable?

Reliability for examiner and examinee is a critical issue and depends on the availability of (information) technology - e.g. computers and computer networking technologies - during the exam [4]. At the time of writing, the SEE depends on the online connection between the SEB and the Moodle Server. In case of network failure, none of the users can save current results or proceed to the next question. Thus, the temporary storage of the answers (during network failures) remains a problem. Fortunately, Moodle saves the last answer received and the progress of each examinee. Therefore, the examinee is allowed to continue the exam from the point where the error occurred after potential network problems are solved. In the worst case scenario, the last answer of the examinee is lost. Hardware failures of laptops are not a problem because all answers provided up to the failure would have been stored on the server and the student can simply continue his or her exam on one of our loan devices.

In the context of testing, archiving exams is another important aspect. According to Austrian legislation [15], documents related to written exams have to be archived for at least six months [16], whereas protocols of oral exams have to be archived for at least one year [16]. Moodle, however, offers a practical solution as it automatically archives exams, which dramatically reduces the physical storage requirements and, as a positive environmental side-effect, the amount of paper needed (especially in the case of no-shows).

2.3 How do we maximize the availability of the SEE?

The availability of the SEE can be affected by hardware failure, network drop outs or service outages.

pocket-sized WiFi jammer could be mounted close to or in the room where the eExams take place.

To ensure the maximum stability of the network system, the network department of our university provides high redundancy within the network-core, distribution-switches, firewalls and the border-router as well as load sharing with the Border Gateway Protocol (BGP) in a multihomed environment and redundant cables. The load is separated among different virtual networks. The equipment in use in the core and distribution layer are high-end-components. Furthermore, an uninterruptible power supply is guaranteed during eExams through a diesel generator.

In order to reach the high availability targets of our SEE, we started to monitor all hardware components and services involved in the eExam process. Drop outs of components or services or deviations from thresholds within defined time intervals result in alerts, allowing support staff to react to and resolve issues immediately, leading to crucial time-savings within the identification process of failures, a very high level of availability and continuous system optimization. Monitored components and services include availability of the SEE-servers (implemented with centOS) including CPU, storage as well as DHCP, NFS, TFTP and HTTP services; availability of the administration backend of the SEE including the corresponding HTTP service; availability of Moodle including HTTP-access, as well as end-to-end-tests in the lecture halls with minimal computers (Raspberry Pi); availability of the network (connection between SEE-server, clients and Moodle), and end-to-end performance tests within the network with probes (Raspberry Pi).

2.4 How do we protect the privacy of examinees?

The answers provided by the students as well as the grades they receive are private and must be protected as stipulated by the Austrian Data Protection Act [15]. Subsequently, we use encrypted and authenticated transmission lines between the SEB and the Moodle server (HTTPS) and the Moodle server uses login/password authentication for students and lecturers to ensure that access is only provided to authorized persons.

2.5 How do we support flexibility for examinees?

One service for students, which followed from the development of the SEE, are so called slotted eExams. For the execution of eExams with the SEE, we developed an online-process to register for an eExam some time before the test takes place as well as an online-registration process right before the exam in the lecture hall. Thus, exams, registration data as well as access rights are available online. These processes enabled us to offer several time-slots for an eExam within a week, from which students can freely choose when they want or are able to take an exam. Especially for students who are employed next to their studies, who need to foster children or relatives or whose mobility is restricted, this service is very helpful.

3. Technical Obstacles and Challenges

One of the current restrictions of online exams is the required network connection. WLAN is still prone to failure, especially for larger groups of students. This results into another challenging aspect, namely that lecture halls require LAN and power sockets at least every second seat. Unfortunately, not all lecture halls fulfill these requirements and retrofitting is extremely expensive. The obstacle with the LAN sockets could be overcome with access points. However, using a device only with battery during exams is too risky.

A persistent challenge are new generations of students’ laptops, requiring continuous adaptation of the SEE. For example, UEFI as a new interface between hardware and OS was rather complex. Moreover, some manufacturers started to disable the possibility of PXE-, resp. Net-Boots on their devices, forcing us to find workarounds. Furthermore, since a lot of new laptops come without Ethernet-sockets, we needed to integrate booting via adapters into the SEE.

4. Organizational Obstacles and Challenges

When it comes to eExams, organization is crucial: on the one hand, it is important for users to easily get through the process with the guarantee that everything will run smoothly, and, on the other hand, technological gaps need to be filled. Moreover, only a limited number of lecture halls is suitable for online exams for reasons mentioned above. Thus, lecturers have to book them early. Furthermore, it cannot be assumed that all students have a portable device. Loan devices must be ready for those students and in the case of technical problems or breakdowns of the students’ own devices during the exam. The AAU currently has approximately 100 laptops serving as loan devices for students.

4.1 Organizational challenges before an exam

Students can come to information days, where we change the boot-order of their devices to allow PXE-, resp. NetBoot. Thus, students get a briefing on how an eExam works before the actual exam preventing unnecessary stress.

Lecturers need support when executing online exams as well. In the preparation phase lecturers (in our case) can rely on the support of specially trained eTutors, who help with setting up a test or in choosing pedagogically suitable examination designs.

4.2 Organizational challenges in the lecture hall

Technical support is provided for students before and throughout the examination. In case of a hardware failure, students get a preinstalled loan device and can continue the eExam immediately. Right after the exam it is a great challenge for the organizational staff to get back sound LAN-cables and loan devices. Students sometimes tend to remove the cables with not enough care, ending up damaging them.

4.3 Support after an eExam

Lecturers get support from the eLearning department after an online exam with the evaluating process or with a statistical analysis of questions.

5. Experiences with eExams at the AAU and Further

Developments

In June 2011 we began offering online exams with the SEE. Table 1 shows the growth of eExams conducted with the SEE at the AAU over the last six years.

Table 1. The progression of eExams with the Secure Exam Environment (SEE), * in process

2011 2012 2013 2014 2015 2016 2017* Total

288 2,717 7,475 7,082 8,954 10,391 9,435 46,342

5.1 (Dis-)Advantages for students

In 2012 we surveyed students about the benefits of eExams as well as any obstacles, and technical problems encountered. We received 308 usable questionnaires from a total of 1,075 students. Results showed a positive attitude towards online exams across all four faculties of the AAU.

The majority of the students who took the survey claimed that the key benefits of online assessment include: rapid exam results, improved readability of free-text answers, improved structure and ability to revise answers, and noticeable time savings. Students also found eExams particularly interesting, convenient and in general better than paper-and-pencil exams. Some students did not see any difference compared with conventional testing methods. Students also appreciated the novelty associated with online testing, including its environmentally friendly nature (less paper and printing costs) and the fact that their hands got less tired than during a handwritten exam.

In addition to the survey, we noticed that students at our university usually do not make use of their right to personally contact the lecturer to ask for more valuable feedback on the exam results [5][2]. This hurdle can be overcome with automated or individual feedback in eExams. The Moodle login-data shows that all students seize the chance to view feedback without the need for personal consultation instead. Moreover, eExams at the AAU are also used for self-testing to give students the opportunity to receive feedback on their performance (see also [5]).

5.2 (Dis-)Advantages for lecturers

Lecturers commonly mention the automated evaluation of standardized questions and the convenience of avoiding hand-written exams as clear advantages of the SEE. Furthermore, questions can easily be created, stored, extended, organized, and reused for future exams, fostering sustainability and decreasing work load. Thus, the time savings can be devoted to new topics or in-depth discussion [5]. Since lecturers at the AAU can freely choose which type of exam to offer to their students, solely those lecturers who see more advantages than disadvantages in eExams use them.

5.3 Further developments

Currently, we are only able to execute one eExam with specific settings, e.g. additional software, at the same time. Therefore, we are developing a boot environment which enables us to run eExams with various additional software simultaneously by recognizing the identity of the student and transmitting the proper exam environment, even in different lecture halls at the same time. Furthermore, the support of newer devices without LAN ports is in development. However, for the future it is planned to provide a WLAN access point. Finally, like every software solution, the SEE needs constant security and compatibility updates.

6. Conclusion

eExams extend the possibilities for assessment in terms of quality and efficiency. Better alignment in teaching, the opportunity to use additional software, reduced possibilities for cheating, and reduced effort for correcting exams are some of the many advantages. Nevertheless, they are only one of a variety of methods for assessing performance and giving students feedback on their learning progress.

References

1. Müller, F. H., Bayer C.: Prüfungen: Vorbereitung - Durchführung - Bewertung.

Förderung von Kompetenzen in der Hochschullehre. Hawelka, B., Hammerl, M., Gruber, H. (eds.): 223-237. Asanger, Kröning (2007).

2. Marriott, P.: Students' Evaluation of the Use of Online Summative Assessment on an Undergraduate Financial Accounting Module. British Journal of Educational Technology, vol. 40 Issue 2, pp. 237-254 (2009).

3. Gibson, D. C., Webb, Mary, E.: Data science in educational assessment. Education and Information Technologies ( 20, Issue 4), 697–71 (2015).

4. Hewson, C.: Can Online Course-Based Assessment Methods Be Fair and Equitable? Relationships between Students' Preferences and Performance within Online and Offline Assessments. Journal of Computer Assisted Learning, (2012).

5. Anakwe, B.: Comparison of Student Performance in Paper-Based Versus Computer-Based Testing. Journal of Education for Business (October), 13-18 (2008).

6. Fluck, A. (2011) eExaminations Strategic Project Final Report for Academic Senate, University of Tasmania. Meeting 1/2011, cited in Fluck, A., Hillier, M.: Innovative assessment with eExams. Paper presented at the Australian Council for Computers in

Education (ACCE) conference, 29 September – 2 October, Brisbane, Queensland (2016).

7. Biggs, J., Tang, C.: Teaching for Quality Learning at University. McGraw Hill, Berkshire (2011).

8. Müller, A., Schmidt, B.: Prüfungen als Lernchance: Sinn, Ziele und Formen von Hochschulprüfungen. Zeitschrift für Hochschulentwicklung (4 No.1), 23-45 (2009). 9. Sharpe, R., Oliver, M. Designing courses for e-learning. In: Beetham, H., Sharpe, R.

(eds.): Rethinking Pedagogy for a Digital Age. Designing and delivering e-learning. Routledge, London and New York (2007).

10. Frankl, G., Schartner, P., Zebeding, G.: The "Secure Exam Environment" for Online Testing at the Alpen-Adria-Universität Klagenfurt/Austria. In: World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education. Association for the Advancement of Computing in Education (AACE), Hawaii (2011).

11. SoftwareSecure, http://www.softwaresecure.com/, last accessed 2017/08/16. 12. Safe Exam Browser (SEB), http://www.safeexambrowser.org/news_en.html, last

accessed 2017/08/16.

13. Virtual Box, http://ww.virtualbox.org, last accessed 2017/08/16 14. Bundeskanzleramt Österreich,

http://www.ris.bka.gv.at/GeltendeFassung.wxe?Abfrage=Bundesnormen&Gesetzesnumm er=20002128, last accessed 2017/08/16.

15. Bundeskanzleramt Österreich,