Online Education Quality Assessment in Higher School

(1)

Online Education Quality Assessment in Higher School

^*

Anatoliy V. Sigal [0000-0003-2090-4464], Maria A. Bakumenko and Dmitriy V. Titarenko [0000-0001-6883-726X]

V.I. Vernadsky Crimean Federal University, Sevastopolskaya Str. 21/4, 295015 Simferopol, Russia

bakumenko_ma@mail.ru

Abstract. E-learning is essential in some instances and, with the right approach, can produce a high level of knowledge. With the growing demand for e-learning, the problem of online education quality assessment is becoming increasingly important. The paper singles out the advantages and disadvantages of online education for students, education institutions and employers, and analyses approaches to online education quality assessment in institutions of higher education. This assessment should be carried out based on a scientific methodology using modern tools of economic and mathematical modeling and information technology. The authors suggest that the quality of online education shall be evaluated from the standpoint of the project approach, with due regard to commercial, reputation, and strategic aspects, i. e. regarding an online education program (or a set of programs) as an investment project and assessing the efficiency of the given project for its core participants – the higher education institution and its students. The obtained performance indicators will constitute the objective assessment of the quality of e-learning in a particular higher education institution. These indicators are the response of the socio-economic system to the quality of online education.

To estimate the values of the weight coefficients that define the importance of the opinions of target groups, the authors propose using a scheme based on Fishburn sequences. They also note that it is feasible to design a software application for assessing the quality of online education.

Keywords: E-learning, Online Education, Quality, Assessment Criterion, Higher Education Institution, Economic and Mathematical Modelling, Information Technology, Investment Project, Efficiency, Student, Fishburn Sequences, Soft- ware Application.

1 Introduction

A major challenge faced by most of the world’s economies is how to improve the quality of management and the quality of produced goods and services to boost their competitiveness in the global market. However, it is impossible to meet the challenge without generating a higher return from education across the board and especially in the

*

(2)

field of higher education. Knowledge is the catalyst for the development of a modern economy and the main factor of production.

One of the forms of training is e-learning, which is now becoming more and more popular. According to P. Ginns and R.A. Ellis, “e-learning is a growing and important part of student experiences of learning at a university internationally” [1]. Q.N. Naveed et al. say that “the E-Learning usage is growing rapidly and being preferred over the conventional teaching-learning process in a big way” [2].

E-learning is characterized by both positive and negative sides. Recently, the quality of online education has been the subject of much debate. Nevertheless, e-learning is essential in some instances (for example, during COVID-19 or for physically challenged people) and, with the right approach, can produce a high level of knowledge (for example, a second university degree for employed and highly motivated people). “As the recent coronavirus outbreak prompted universities to start shifting classes online either for a few weeks or for the remainder of the spring semester of 2020, e-learning and remote education have popped up as the magical alternative for in-person classes in the time of the COVID-19 pandemic” [3].

It should be noted that nowadays specialists are facing the need for lifelong learning, which is due to the high dynamism of the professional environment. To keep their com- petitive edge in the labor market, modern specialists with a degree have constantly to enhance and expand their knowledge, skills, and abilities. And, in the long run, this trend is likely to intensify further, while e-learning is the best choice for working specialists.

With the growing demand for e-learning, the problem of online education quality assessment is becoming increasingly important. This assessment should be carried out based on a scientific methodology using modern tools of economic and mathematical modeling and information technology since information technology has become a seri- ous decision support tool.

A significant number of scientific works are devoted to the problem of assessing the quality of online education. For example, different approaches to identifying criteria for online education quality assessment can be found in the works of the following authors: D. Al-Fraihat, M. Joy, R. Masa'deh and J. Sinclair [4], A.I. Guseva and E.B. Vesna [5], D. Masoumi and B. Lindström [6], S.Y. Sergeeva and E.D. Obrevko [7].

Despite a significant number of works on the problems of assessing the quality of online education, this research area still requires attention. We believe that there is a need to create a methodology for assessing the quality of online education, with objective and unbiased evaluation.

The objective of the paper is to analyze approaches to online education quality assessment in higher school and to propose an original approach to address the issue, which should form the basis for creating a software application for online education quality assessment in higher school.

(3)

2 The Advantages and Disadvantages of Online Education

Currently, institutions of higher education are introducing e-learning to expand their markets, reduce costs, and achieve more flexibility. E-learning provides its participants with several undeniable benefits, but it is not free of some essential drawbacks given in Table 1.

Table 1. The advantages and disadvantages of online education for students, education institu- tions, and employers.

Parties Advantages Disadvantages

Stu- dents

Time-saving (for example, for transpor- tation); cost-saving (transport and other expenditures); the possibility to combine work and learning; the accessibility of learning materials 24/7; the availability of education for physically challenged people; the possibility of individualiza- tion of the trajectory and content of learning.

Difficulty in choosing a training program;

learning materials may be of low quality;

during e-learning, difficulties may arise with understanding the material, especially it's a practical part; students’ low motivation can be a constraining factor for obtaining the necessary knowledge, skills, and abilities; insuf- ficient level of social communication skills.

Edu- cation institutions

The possibility to expand the market;

broad market coverage; reduction of infrastructure maintenance costs; the possibility to collaborate with other education establishments, including foreign ones.

The problem of motivating students to com- plete the full cycle of training; the problem of checking the level of knowledge, skills, and abilities of students; the problem of confirm- ing the identity of students during learning and checking the acquired knowledge; high labor intensity of development of training courses.

Em- ploy- ers

The possibility of employee training on the job.

Lack of a clear understanding of the presence and level of graduates’ competencies.

a Designed by the authors partly using the materials [5; 8].

3 Analyzing Approaches to Online Education Quality Assessment in Higher School

“E-Learning (EL) is explained as a network affinity group sharing their information, knowledge, proficiency, and conferring education to many learners geographically in the same or diversified. EL is learning and teaching online and sharing resources elec- tronically” [9]. “Quality in education can be understood as a set of attributes or charac- teristics, selected to evaluate the service, which affects customer satisfaction, either ex- plicitly or implicitly” [10].

(4)

Online education quality assessment is a complex task; this is due to its qualitative nature, which, in some instances, may lead to a certain degree of subjectivity. The problem of assessing the quality of e-learning is complex and subjective. The success of this task will be determined by the comprehensiveness of the approach and the correct choice of criteria and assessment methods.

Table 2. Online education quality assessment criteria.

Author(s) Criteria Al-Fraihat D.

et al.

- technical system quality;

- information quality;

- service quality;

- educational system quality;

- support system quality;

- learner quality;

- instructor quality [4].

Shee D.Y., Wang Y.S.

- learner interface;

- learning community;

- system content;

- personalization [11].

Masoumi D.,

Lindström B. - institutional factor (institutional affairs, administrative affairs, research, reputation);

- technological factor (development and sustainability of technological infrastructure, the functionality of technological infrastructure, accessibility, reusability, interface design);

- instructional design factor (clarifying expectations, personalization, se- lecting proper learning scenarios, organizing learning resources, cur- rency, and accuracy of learning resources);

- pedagogical factor (student-centredness, communication, and interac- tivity, social aspects, learning environments, assessment, learning resources);

- student support (administrative support, technical support);

-teachers support (technical assistance in course development, administrative support, pedagogical support);

- evaluation factor (cost-effectiveness, learning effectiveness, student satisfaction, teachers’ satisfaction) [6].

Guseva A.I., Vesna E.B.

- performance indicators (indicators of achieving the objectives of the implementation of online programs; indicators of evaluating the quality of the implementation of programs);

- efficiency indicators (indicators of the popularity of online programs;

indicators of the usability of online resources; indicators of assessing the quality of graduates’ competence level) [5].

Sergeeva S.Y., Obrevko E.D.

- criteria of the quality of the result of the educational process;

- criteria of the quality of conditions for the implementation of the educational process;

- criteria of the quality of the implementation of the educational process [7].

(5)

This case requires the application of a comprehensive approach. It is also necessary to take into account that changes in the external environment may require adjustments to the designed assessment algorithm. Over time, the assessment criteria should be ad- justed and adapted to the requirements of the current period.

As M.G. Sergeeva notes, the quality of education including e-learning “... does not have a generally accepted formalized description and a particular set of criteria that characterize its essence” [12]. The paper [7] gives a comparative analysis of education quality assessment criteria in the EU (European Quality Improvement System, EQUIS methodology), the US (Association to Advance Collegiate Schools of Business, AACSB methodology), and Russia. Table 2 contains various approaches to the selec- tion of criteria for assessing the quality of online education, presented in the scientific literature.

4 Evaluation of the Quality of E-Learning Using the Project- Based Approach

The authors believe that in a higher education institution the quality of e-learning as a whole or a particular online education program can be evaluated using the project-based approach, i. e. regarding an online education program (or a set of programs) as an investment project and assessing the efficiency of the given project for its core participants – the higher education institution and its students. In this case, we are talking about calculating the commercial efficiency of the project, which is taking into account the benefits and costs of its core participants.

The obtained performance indicators will constitute the objective assessment of the quality of e-learning in a particular higher education institution. These indicators are the response of the socio-economic system to the quality of online education. See Fig- ure 1.

In the classic sense, the efficiency of an investment project is a category that expresses “... the compliance of the results and costs of the project with the goals and interests of its participants including, if necessary, the state and the population” [13].

The essence of the process of assessing the efficiency of an investment project is to compare its results and costs. The results and costs of an online education project will be different for each participant and can be characterized not only by a quantitative nature (for example, financial) but also by a qualitative (intangible) nature. Online education performance assessment requires the evaluation of all project outputs, both tan- gible and intangible. According to leading Russian experts, the main criterion of the efficiency of an investment project is Net Present Value (NPV) [13].

In the authors’ opinion, when assessing the commercial efficiency of an online education project for its core participants, it is feasible to apply the concept of evaluating the commercial efficiency of investment projects in the real economy, proposed in the monograph by M.A. Bakumenko [14]. This concept singles out three components of project efficiency evaluation: commercial, strategic, and reputation.

(6)

Fig. 1. The formation and evaluation of the response of the socio-economic system to the qual- ity of online education (OE) in a higher education institution.

In the case of assessing the commercial efficiency of an online education project for its participants:

 the commercial component (СС) shows how a project participant achieves their fi- nancial goals during the project’s design period;

 the strategic component (SC) shows how an online education project affects the stra- tegic development of a project participant;

 the reputation component (RC) reflects a change in the reputation of a project par- ticipant, resulted from participation in the online education project. See Figure 2.

Fig. 2. Commercial results of online education (OE) project participants.

(7)

5 Methodology for Assessing the Quality of Online Education

The essence of the methodology for assessing the quality of online education, proposed by the authors, is briefly described in Table 3.

Following the proposed approach, it is possible to talk about high-quality e-learning in a higher education institution subject to the conditions as follows:

1. 𝐶𝐶_𝑖≥ 0, i1,2. I. e. the commercial component for all online education project participants should be non-negative.

2. The online education project will have a positive impact on the process of strategic development of the higher education institution and the student.

3. RC_i 0, i1,2. I. e. the online education project will have a positive impact on the reputation of the higher education institution and the reputation of the student (graduate) as a specialist in a particular subject area.

The reputation component of the proposed methodology (for a particular online education project participant) can be determined by equation (1), subject to the constraints (2).

 





 

 ^T

t

t t

t r w

r C

R

1

, (1)

1





 T

t

wt , 𝑤_𝑡≥ 0, t1,T. (2) Where RC is a change in the image (reputation) of the OE project participant, resulting from participation in the OE project; r_t^ is the response of t target group to the OE project participant after project implementation; r_t is the response of t target group to the OE project participant before project implementation; w_t is a weight coefficient that defines the importance of the opinion of t target group for the OE project participant; T is the number of target groups.

The values of r_t and r_t^ are in the range of [– 1; 1] and correspond to a special scale that numerically expresses the attitude of a certain target group towards the OE project participant. For this purpose, one can apply the scale presented in [14; 15]. To define the values of r_t and r_t^, one can use expert technologies combined with qualitative research methods (media monitoring, survey, questionnaire, business conversation, etc.).

If RC > 0, the reputation of the OE project participant has improved after participation in the project; if RC < 0, the reputation of the OE project participant has deterio- rated after participation in the project; if RC  0, the reputation of the OE project participant has remained the same.

(8)

Table 3. The basic concepts of the methodology of online education (OE) quality assessment based on the project approach. ^b

Indica- tor

Online education project participant

Higher education institution E-learner

Com- mer- cial component (СС)

CC1 is the value of NPV, reflecting the commercial efficiency of a particular OE program (or a set of programs) over the project’s design period for the higher education institution.

The project’s design period equals the duration of learning through a particular OE program (if several education programs are included in the analysis, the longest training interval is taken as the design period).

We are talking about efficiency calculations in the operating phase of the project life cycle. The information base for calculations is the data provided by the financial service of the higher education institution. The process of calculations takes into account the benefits and costs of the higher education institution, determined by a particular OE program (or their to- tal).

The discount rate is determined by the available alternative investment opportu- nities following the modern methodology for assessing the efficiency of investment projects.

CC2 is the expected value of NPV, reflecting the commercial efficiency of a particular OE program for some hypo- thetical (average) student.

The project’s design period equals the sum of the duration of learning through a particular OE program and five years af- terward (related to the possibility of put- ting the obtained knowledge, skills, and abilities into practice as well as to the rel- evance of this knowledge).

To find the expected value of NPV, the scenario analysis method should be applied. Based on the processing of statistical information and using expert technologies, the cash flows of three scenarios are predicted: optimistic, basic, and pes- simistic ones, and the probability of oc- currence is determined for each scenario.

The process of calculations takes into account the benefits and costs of the student over the project’s design period, due to participation in a particular OE program (including opportunity cost).

The discount rate is determined by the available alternative investment opportu- nities.

Strate- gic component (SC)

SC1 characterizes the degree to which the OE project encourages the implementation of potential projects in the future (the technique presented in the work [16]

is applied).

SC2 characterizes the possibility of the graduate to advance in the long run. It is defined by experts based on the available statistics. The analog method can also be used here.

Reputa- tion component (RC)

RC1 characterizes a change in the reputation of the higher education institution, resulted from participation in the OE project.

RC2 characterizes a change in the reputation of the worker (specialist), resulted from participation in the OE project.

b Formed by the authors partly using the materials [14; 16].

(9)

The reputation of a higher education institution (as well as the reputation of a specialist) is one of the most significant intangible assets and strongly influences the process of their strategic development. To preserve (improve) reputation, it is very important to adhere to ethical principles in one’s activities. An organization’s ethical behavior will enhance its competitiveness (Table 3).

The paper [17] shows that the ethical behavior of firms boosts competitiveness not only of the corporation (firm) but also of the national economy.

To estimate the values of the weight coefficients that define the importance of the opinions of target groups, the authors propose using a scheme based on Fishburn sequences.

6 Fishburn Sequence-Based Scheme to Estimate the Values of Weight Coefficients

Notably, in equation (1), it is crucial correctly to estimate the unknown values of w_t components of the weight vector. The search for these estimates can be based on various methods and models, including the concept of applying statistical games combined with antagonistic games, described in detail in the monograph by A.V. Sigal [18].

Let us take a closer look at the methodology for estimating the values of w_t weight coefficients, based on Fishburn sequences [19], first of all, their particular case − generalized Fishburn progressions (see for example [19]).

Without loss of generality of reasoning, we can assume that the target groups under consideration are arranged in descending order in terms of the importance of their opinions (as seen by a certain OE project participant). This indicates that the values of the weight coefficients generate the non-ascending sequence: w 1w 2…w T. Thus, on the components of the weight vector, one or another linear order relation is set by the subjective preferences of a certain OE project participant. Linear order relations were studied in detail by Peter C. Fishburn and are given, for example, in the monograph by R. I. Trukhaev [20]. Let us give definitions of the two most common types of linear order relations (given the case of non-ascending sequences) [20].

A simple linear order relation refers to the relations w 1w 2…w T. A partially strengthened linear order relation refers to the relations w_tw₁...w_t_₁, t2,T.

Let us set the weight vector W(w 1; w 2;…; w T).The vector’s w_t components shall satisfy all the constraints (2). What is more, as the target groups under consideration are arranged in descending order in terms of the importance of their opinions, the values of the weight coefficients satisfy the inequalities w 1w 2…w T.

If in the subjective view of a certain OE project participant, a simple linear order relation holds for the unknown values of the weight coefficients, Peter C. Fishburn sug- gests that the estimates of w_t components generate an arithmetic progression, and if a partially strengthened linear order relation holds, then they form a monotonic geometric progression (see for example [20]). The corresponding Fishburn’s equations can be

(10)

easily generalized in the case of monotonic progressions, which leads to the notion of generalized Fishburn progressions (see for example [19]).

We call generalized Fishburn progressions the progressions {w 1; w 2;…; w T}, sat- isfying all the constraints (2): generalized arithmetic Fishburn progressions are arithmetic progressions that satisfy all the constraints (2), generalized geometric Fishburn progressions are geometric progressions that satisfy all the constraints (2).

It can be easily proved that generalized arithmetic Fishburn progressions represent arithmetic progressions of the form

     

T

x t T x T

x t T w_t T











 





 

 

 2

1 2 1 2

2 1

1 , t1,T,

whose difference satisfies the inequality



1



2



  T

x T ,

while generalized geometric Fishburn progressions represent geometric progressions of the form

1 1

1 1 1

1 _  _



 

 

 _T ^t _T ^t

t x

x x x

x

w x , t1,T,

whose common ratio satisfies the inequality x0. For example, the basic properties of generalized Fishburn progressions are given in the monograph [19]. Further gener- alization of the concept of “generalized Fishburn progressions” leads to the concept of Fishburn sequences [19].

Monotonic sequences {w 1; w 2;…; w T} that satisfy all the constraints (2) shall be regarded as Fishburn sequences.

We can estimate the unknown values of w_t weight coefficients according to the following three-step scheme based on Fishburn sequences.

Step 1. To define what type of linear order relations the unknown values of w_t weight coefficients shall satisfy.

Step 2. To choose a convenient monotonic sequence {a 1; a 2;…; a T} of nonnegative numbers, the sum of which is a positive number, and the sequence itself satisfies the type of linear order relations that the unknown values of w_t weight coefficients shall satisfy. Such a sequence {a 1; a 2;…; a T} can be referred to as a sequence generating a Fishburn sequence.

Step 3. To estimate the unknown values of wt weight coefficients by the elements of the corresponding Fishburn sequence by the equation





 _T

j j t t

a w a

1

, t1,T .

Notably, as a sequence generating the corresponding Fishburn sequence, one can choose monotonic progressions of natural numbers, including {a 1; a 2;…; a T}{1; 1;…; 1} or {a 1; a 2;…; a T}{T; T–1;…; 1}, as well as such sequences of natural numbers as Fibonacci numbers, Euclid numbers, Mersenne numbers,

(11)

Fermat numbers (see for example [19]), etc. An arithmetic progression of natural numbers generates a generalized arithmetic Fishburn progression, while a monotonic geometric progression of natural numbers generates a generalized geometric Fishburn progression. It should be noted that any generalized arithmetic Fishburn progression al- ways satisfies the corresponding simple linear order relation, while a generalized geometric Fishburn progression satisfies the corresponding partially strengthened linear order relation only if specified requirements hold for the common ratio of the progression (see for example [19]).

Having estimated the unknown values of w_t weight coefficients by the elements of the selected Fishburn sequence, one should calculate the desired change in the image (reputation) of the OE project participant, resulted from participation in the OE project, applying the elements of the selected Fishburn sequence to the equation (1) to calculate the RC indicator.

7 Concerning the Creation of a Software Application for Online Education Quality Assessment in Higher School

At present, information technology has become an integral part of management and managerial decision-making. According to the authors, the proposed methodology for assessing the quality of online education in higher schools should form the basis of the corresponding software application, since all the calculations within the proposed methodology are rather time-consuming.

Note that the commercial component of the proposed methodology can be found for each online education project participant using special software designed to assess the effectiveness of investment projects and to manage them. Examples of such software solutions well established in the market are Primavera and Microsoft Project, a comparative analysis of which is given, in particular, in the paper [21]. Aided by these software solutions, one can plan cash flows for each online education project participant and find the value of NPV or the expected value of NPV.

However, the proposed methodology has two more important specific components:

the strategic component and the reputation one. The presence of these components in the proposed methodology necessitates the development of an appropriate software application that would facilitate all the calculations.

One of the good tools for modeling software applications is the unified modeling language (UML). See, for example, the works [22, 23]. Moreover, according to B. Dob- ing and J. Parsons, “UML should not be considered exclusively as a language for software professionals; a greater understanding of UML diagrams and their roles in build- ing systems is needed throughout organizations” [22].

Figure 3 contains a use case diagram that describes the main functionality of the software application proposed for design. “Use case is a high-stage description of what the approach is meant to do, whose purpose is to capture the approach requirements” [23].

(12)

Figure 3 shows that the software application proposed for design should assess the quality of online education in a particular higher education institution, and (when necessary) conduct a similar comparative analysis for several educational institutions.

8 Conclusions

E-learning provides its participants with several undeniable benefits, but it is not free of some essential drawbacks.

With the growing demand for e-learning, the problem of online education quality assessment is becoming increasingly important. This assessment should be carried out based on a scientific methodology using modern tools of economic and mathematical modeling and information technology.

Fig. 3. Use Case diagram of assessing the quality of online education (ОЕ) in higher school The authors believe that in a higher education institution the quality of e-learning as a whole or a particular online education program can be evaluated using the project- based approach, i. e. regarding an online education program (or a set of programs) as an investment project and assessing the efficiency of the given project for its core participants – the higher education institution and its students. The obtained performance indicators will constitute the objective assessment of the quality of e-learning in a particular higher education institution. These indicators are the response of the socio-economic system to the quality of online education.

Following the proposed approach, it is possible to talk about high-quality e-learning in a higher education institution subject to the conditions as follows: the commercial component for all online education project participants should be non-negative; the

(13)

online education project will have a positive impact on the process of strategic development of the higher education institution and the student; the online education project will have a positive impact on the reputation of the higher education institution and the reputation of the student (graduate) as a specialist in a particular subject area.

To estimate the values of the weight coefficients that define the importance of the opinions of target groups, the authors propose using a scheme based on Fishburn sequences.

The proposed methodology for assessing the quality of online education in higher schools should form the basis of the corresponding software application since all the calculations within the proposed methodology are rather time-consuming.

We believe that the proposed approach to assessing the quality of online education is reasonable. This approach can add to the existing practices for evaluating online education quality.

The study has been carried out with partial financial support from the Russian Foun- dation for Basic Research, project No 18-010-00688.

References

1. Ginns, P., Ellis, R.A.: Evaluating the Quality of E-Learning at the Degree Level in the Stu- dent Experience of Blended Learning. British Journal of Educational Technology 40(4), 652–663 (2009), DOI:10.1111/j.1467-8535.2008.00861.x.

2. Naveed, Q.N. et al.: Evaluating and Ranking Cloud-Based E-Learning Critical Success Fac- tors (CSFs) Using a Combinatorial Approach. IEEE Access 7, 157145–157157 (2019), DOI:

10.1109/ACCESS.2019.2949044.

3. Muhammad, A.H. et al.: A Hierarchical Model to Evaluate the Quality of Web-Based E- Learning Systems. Sustainability 12 (2020), https://doi.org/10.3390/su12104071.

4. Al-Fraihat, D., Joy, M., Masa'deh, R., Sinclair, J.: Evaluating E-Learning Systems Success:

an Empirical Study. Computers in Human Behavior 102, 67–86 (2020), https://doi.org/10.1016/j.chb.2019.08.004.

5. Guseva, А.I., Vesna, Е.B.: Evaluation of the Performance and Effectiveness of Online Edu- cation Programs. Contemporary Issues of Science and Education 6 (2013), http://science- education.ru/ru/article/view?id=11000, last accessed 2020/09/01. (in Russian)

6. Masoumi, D., Lindström, B.: Quality in E-Learning: a Framework for Promoting and As- suring Quality in Virtual Institutions. Journal of Computer Assisted Learning 28, 27–41 (2012), DOI: 10.1111/j.1365-2729.2011.00440.x.

7. Sergeeva, S.Y., Obrevko, Е.D.: Modern Approaches and Methods of Education Quality As- sessment. Young Scientist 37(275), 162–165 (2019). (in Russian)

8. Bukeikhanov, N.R., Gvozdkova, S.I.: Evaluation of the Effectiveness of Digital Teaching Technologies in the Context of СOVID-19. Russian Regions: A Look into the Future (2020), https://cyberleninka.ru/article/n/otsenka-effektivnosti-tsifrovyh-tehnologiy-prepodavaniya- v-usloviyah-sovid-19, last accessed 2020/09/01. (in Russian)

9. Naim, A., Alahmari, F.: Reference Model of E-Learning and Quality to Establish Interoper- ability in Higher Education Systems. International Journal of Emerging Technologies in Learning 15(2) (2020), https://doi.org/10.3991/ijet.v15i02.11605, last accessed 2020/09/01.

(14)

10. Vagarinho, J.P., Llamas-Nistal, M.: Process-Oriented Quality in E-Learning: a Proposal for a Global Model. IEEE Access 8, 13710–13734 (2020), DOI:

10.1109/ACCESS.2020.2965619.

11. Shee, D.Y., Wang, Y.S.: Multi-Criteria Evaluation of the Web-Based E-Learning System: a Methodology Based on Learner Satisfaction and Its Applications. Computers & Education, 50, 894–905 (2008), DOI: 10.1016/j.compedu.2006.09.005.

12. Sergeeva, М. G.: Ensuring the Quality of Learning Through Digitalization of Education.

Pedagogy Issues 3, 262–265 (2019). (in Russian)

13. Vilensky, P.L., Livshits, V.N., Smolyak, S.А.: Investment Project Efficiency Evaluation:

Theory and Practice. Delo, Moscow (2004). (in Russian)

14. Bakumenko, M.A.: Evaluation of Commercial Effectiveness of Real Investment Projects:

Methods and Models. DIAIPI, Simferopol (2014). (in Russian).

15. Bakumenko, M., Sigal, A.: Reputation Aspects in Investment Decision Making: Contribu- tion to Corporate Security. International Journal of Risk Assessment and Management 21(1/2), 111–134 (2018), https://doi.org/10.1504/IJRAM.2018.090261.

16. Bakumenko, М.А.: Investment Decision-Making by Corporations Concerning Reputation Aspects. Proceedings of the Crimean Federal University. Economics and Management 4(4), 10−20 (2018). (in Russian)

17. Bakumenko, M., Sigal, A.: Concerning the Evaluation of Ethics Impact on National Econ- omy’s Competitiveness. International Journal of Risk Assessment and Management 23(1), 80−105 (2020), https://doi.org/10.1504/IJRAM.2020.106164.

18. Sigal, А.V.: Games Theory in Economic Decision-Making. DIАIPI, Simferopol (2014). (in Russian)

19. Sigal, А.V., Remesnik, Е.S.: Fishburn Sequences and their Application in the Modern Port- folio Theory: Monograph. Simferopol (2018). (in Russian)

20. Truchaev, R.I.: Decision-making Models in Uncertain Environment. Nauka, Moscow (1981). (in Russian)

21. Wali, K.I., Othman, S.A.: Comparison and Assessment of Using Primavera and Microsoft Project in Construction Projects in Erbil City. ZANCO Journal of Pure and Applied Sci- ences, 31(s3), 285−291 (2019), DOI: http://dx.doi.org/10.21271/ZJPAS.31.s3.39.

22. Dobing, B., Parsons, J.: How UML is Used. Communications of the ACM 49(5), 109−113 (2006).

23. Rajagopal, D., Thilakavalli, K.: A study: UML for OOA and OOD. International Journal of Knowledge Content Development & Technology 7(2), 5−20 (2017) http://dx.doi.org/10.5865/IJKCT.2017.7.2.005, last accessed 2020/09/01.