Combining Content Analytics and Activity Tracking to Identify User Interests and Enable Knowledge Discovery
Andrii Vozniuk, Mar´ıa Jes´ us Rodr´ıguez-Triana, Adrian Holzer, and Denis Gillet REACT Group, EPFL, Station 9, 1015 Lausanne, Switzerland.
Email: {andrii.vozniuk, maria.rodrigueztriana, adrian.holzer, denis.gillet}@epfl.ch
ABSTRACT
Finding relevant content is one of the core activities of users interacting with a content repository, be it knowledge work- ers using an organizational knowledge management system at a workplace or self-regulated learners collaborating in a learning environment. Due to the number of content items stored in such repositories potentially reaching millions or more, and quickly increasing, for the user it can be chal- lenging to find relevant content by browsing or relying on the available search engine.
In this paper, we propose to address the problem by pro- viding content and people recommendations based on user interests, enabling relevant knowledge discovery. To build a user interests profile automatically, we propose an approach combining content analytics and activity tracking. We have implemented the recommender system in Graasp, a knowl- edge management system employed in educational and hu- manitarian domains. The conducted preliminary evaluation demonstrated an ability of the approach to identify interests relevant to the user and to recommend relevant content.
Categories and Subject Descriptors
K.3.1 [Computers and Education]: Computer Uses in Education; H.5.2 [Information interfaces and presen- tation]: User interfaces
Keywords
Learning Analytics, Educational Data Mining, Interests Min- ing, Knowledge Discovery, Recommender System, Content Analytics, Text Mining, Activity Tracking, Information Re- trieval
1. INTRODUCTION
Knowledge plays an essential role in value creation in the post-industrial economy. Knowledge is acquired and en- riched in learning, which often takes place at a workplace or in an educational setting. While learning, people inter- act with content as a knowledge medium, located in various content repositories. In an educational setting, such content repositories are usually learning environments, where both students and teachers interact with the content found there.
Teachers would regularly interact with the content, when preparing a course while students - when following a course or just collaborating with peers.
When working on a course in a learning environment, teachers enrich the system with relevant materials includ- ing text files, web links, videos, audio recordings coming
from their device or the cloud. Other teachers can benefit from content already available in the platform when prepar- ing their courses. Moreover, it may also be beneficial for the students to have access to the content that is relevant to their interests, but which the teacher did not directly include into her course [25]. In the case of learning environments with a vast number of content items, it may be hard for the user to find content items corresponding to her interests.
To address the mentioned issues, we propose to employ a recommender system that combines the content analytics, activity tracking, and information retrieval techniques to (1) build the user interests profile and afterward (2) to suggest content relevant to the user and users with similar interests enabling knowledge discovery. To perform the recommenda- tion, first, for each item available in the content repository, we employ natural language processing techniques to iden- tify a set of concepts related to the content in a similar way how humans would do it. Relying on high-level concepts in- stead of specific words present in the text when constructing user interests profile and afterward finding similar items, al- lows to identify the content that covers the same high-level concepts even if the specific words used in it are different.
Next, we analyse the interactions of the users with the con- tent items based on available user activity recordings and aggregate the concepts in the content that the user inter- acted with building in this way the user interest profile. Fi- nally, we use information retrieval techniques to recommend to the user relevant content based on the similarity between the concepts in the content and concepts identified as user interests. In the same way, our approach allows finding rele- vant users based on the determined interests similarity. Our approach puts the user in control of her interests profile and allows to adjust the interests by removing concepts if nec- essary, as in the case when the user is not interested at the moment in some of the identified concepts.
To evaluate the usefulness and the performance of the algorithm, we have implemented the approach in Graasp, a knowledge management system used in educational [2] and humanitarian [24] settings. Afterward, we have evaluated the approach with teachers, identifying their interests and providing them with recommendations.
This paper describes the algorithm used, the implemen- tation details of the approach in Graasp and the evaluation of the approach with users. The structure of this paper is as follows. First, Section 2 reviews some of the relevant ap- proaches to content analytics, activity tracking, and knowl- edge discovery. Afterward, Section 3 explains our approach to constructing user interests profile and demonstrates how
we make recommendations based on the interests. Section 4 illustrates an implementation of the proposal, while Sec- tion 5 talks about the evaluation methodology and the re- sults. Finally, Section 6 presents the conclusions and high- lights directions for the future work.
2. RELATED WORK
In this section, we review relevant work from the domains of content analytics, activity tracking, user interests mining and take a look at notable systems supporting knowledge discovery.
2.1 Content Analytics and Activity Tracking
Content Analytics. Content analytics allows the ma- chine to gain an understanding of the content, similarly to how a human would do it by, among others, extracting the main topics, concepts, and entities present in the content.
Kovanovic et. al. did an extensive overview of content ana- lytics as one of the often employed techniques in the domain of learning analytics [12]. For instance, in the line of our work, Bosnic et al. proposed to use automatic extraction of keywords from textual content as a foundation for content recommendations [3]. It is worth noting that existing pa- pers focus mainly on analysis of textual content [12], while recent progress in the understanding of multimedia formats, such as object recognition in images or videos [13], or speech recognition allow broadening the scope of content analytics from purely textual information to the various multimedia formats.
Understanding the content alone is not sufficient for un- derstanding the learning since, according to Moore, learner- content interaction is a defining characteristic of education [14].
Moore argues that such learner-content interaction is neces- sary to happen for the education to take place since ”it is the process of intellectually interacting with content that results in changes in the learner’s understanding, the learner’s per- spective, or the cognitive structures of the learner’s mind” [14].
Recognising the importance of the interaction, below we con- sider approaches to capturing and persisting the interactions through activity tracking.
Activity Tracking. User activities tracked by a learning platform is a common data source in the field of learning an- alytics [18, 19]. Usually, a learning management system or a learning environment have a logging infrastructure in place that records how the user interacts with the platform [18].
The more modern educational platforms support a struc- tured representation of user activities using well-defined for- mats including ActivityStreams used in [23], xAPI employed in [11] or IMS Caliper outlined in [19]. On a high-level, all these three formats record user-platform interactions in the form of the actor-verb-object triplet capturing who did what with what on the platform. However, on a more de- tailed level, each format captures additional aspects of the interaction. In the triplet, the verb indicates the type of interaction, for instance, the verb ”accessed” would mean that the user viewed content, ”downloaded” - downloaded the content and so on. Having a common set of verbs with a well-defined meaning is critical for being able to benefit from user interactions captured by several platforms [11].
Combining Both. While there is a considerable num- ber of studies employing content analytics or relying on in- teraction analysis, the number of studies combining both is still somehow limited even taking into account that it is
considered a promising direction [18, 12]. One noticeable recent proposal combining the both approaches is by Kim et. al. [10] where they use content analytics and recorded interaction data to understand better how students learn with video and eventually to improve their experience, for instance by explaining better the identified confusing topics.
Following these recommendations, we consider the combina- tion of both content analytics and activity tracking as a core part of our proposal.
2.2 Mining User Interests
The obtained user interests can be used for different pur- poses, including privacy awareness and recommendations.
Harkous et. al. proposed in [9] to employ a content analy- sis of the files located on Google Drive of a user to under- stand the topics, concepts, and entities relevant to the user.
They used the obtained information with the goal to im- prove the user awareness through a new permissions model called Far-reaching Insights. This model informs the user about the insights that third-party applications can derive about her based on the accessible Drive data given the re- quested permissions are granted. In our approach, we want to explore how identified interests can be used to provide the user with relevant content. In the following subsection, we review some of the systems enabling knowledge discovery with such recommendations.
2.3 Knowledge Discovery Systems
Klamma et al. have formulated a set of requirements for a collaborative adaptive learning platform [16]. One of the requirements is ”Support for personalized learning resource delivery through an intelligent adaptive engine, being able to connect people to the right knowledge and deliver quality learning resources that are tailored to the learner’s prefer- ences and learning goals.” [16]. Learning platforms often integrate such engine in a form of a recommender system.
Drachsler et. al. have conducted an extensive review of 82 recommender systems used to support learning in [5]. Be- low, we take a look at several proposals, particularly relevant to our approach.
Zaldivar et. al. address in [25] the problem of discover- ing by the instructors relevant learning resources used by students when learning, that are not part of the materials provided by the instructor but still can be beneficial for the students. In their approach, the authors record the web pages that students visit and perform a lexical analysis of the page content. Afterward, they apply information re- trieval techniques to identify the online content (webpages) that are the most similar to the content provided by the instructors as part of the course.
In [6] El Helou et. al. proposed a recommender system that considers user interactions with content items to con- struct a user-content associations graph. After the graph is built, the system applies a ranking algorithm to provide the user with personalized recommendations of relevant actors, activity spaces and knowledge assets taking into account the context.
Motivated by the presented approaches, in the next sec- tion, we propose to employ a recommender system that com- bines content analysis, activity tracking to identify user in- terests and information retrieval techniques to suggest rele- vant content and people.
3. INTERESTS-BASED RECOMMENDER
In this section, we explain how our approach works by first automatically identifying user interests and after using the interests to obtain relevant content and people.
3.1 Identifying User Interests
To identify user interests, our system needs, first, to un- derstand the concepts covered in the content. Second, it requires recorded user activities to know how the user inter- acts with the content items. Having both the concepts and the activities, the system can construct the user interests profile. Below, we explain each component of the approach.
Content Analytics. Content can be available in multi- ple formats, and a data processing pipeline needs to be built to extract concepts from the content and after store them in an index for further use. A general representation of the key steps of the pipeline is shown on Figure 1 where different types of content may go through different processing steps to obtain the concepts.
On the first step, textual content is extracted from the stored items. In the second step, the content analysis is performed. For the content analysis, we considered using named entity recognition (NER), concept extraction, and topic modelling. Since NER picks entities only from the words present in the text, using such entities for recommen- dations would limit the discovery only to the content con- taining them directly. Differently, high-level concepts allow identifying relevant content even if the specific words used in it are different. When we applied topic modelling to real data, the identified topics having a high level of abstraction did not seem to capture well the content particularities. For these reasons, we use a set of concepts to describe the con- tent. Finally, on the third step the extracted content and the concepts are tokenized and put into a searchable index so that they can be used on the recommendation step.
Activity Tracking. Our approach requires recording user-content interactions, namely the triplet user-verb-object.
We consider different types of interactions as a manifesta- tion of different interest strength. For instance, intuitively when a user downloads the content it manifests a stronger interest in the content compared to just viewing it online.
Our approach does not assume a specific activity recording technique or data format used, but it requires the approach to capture the user identifier, the verb indicating the type of interaction and, the identifier of the resource the user has interacted with.
Computing User Interests. As the user interacts with the content, the system aggregates the concepts identified in the content, weighting them according to the type of interac- tion as demonstrated on Figure 2. The aggregated concepts constitute the user interests profile.
Let’s look into more details how the system can compute the user interest profiles at any point in time. We denote by nthe number of users on the platform, by p- the number of possible interaction types, bym- the number of content items on the platform and byk - the number of concepts identified in the content. Then at any point in time the user interests profiles can be computed in the following way:
U Cn∗k=
p
X
v=1
wv∗U Avn∗m∗DCm∗k, (1)
where U Cn∗k is the matrix of user concepts of interest,
henceU Cij is the relevance of the conceptscj for the user ui;wv is the weight assigned to specific interaction type v indicating how strongly specific action of the user expresses her interest in the content;U Avn∗m is the matrix capturing user-content interactions of type v, U Avij is the number of times the user ui has done interaction of type v with the contentdj;DCm∗kcontains the concepts represented in the content soDCf∗r is the relevance of conceptcr to the con- tent itemdf.
While the formula presented above is suitable for comput- ing the profile first time when the recommender is deployed, the profile does not need to be recomputed from scratch and can be updated incrementally. On every user-content inter- action, we update in real-time the user concepts of interest based on the ones that were found in the content as follows:
U C1∗kaf ter=U C1∗kbef ore+wv∗U Avn∗m∗DCm∗1, (2) whereU C1∗kbef ore is the vector of user concepts before the interaction andU C1∗kaf ter - after the interaction;U Avn∗mis a matrix having 1 in position (i, j) if the useruihad interac- tion of typev with the content itemdj, all other elements are 0; and DCm∗1 contains relevance values for the item concepts.
Once the profile constructed, in the next section we ex- plain how it can be used for recommendations.
3.2 Recommending Relevant Content and Users
Connecting right people with right knowledge is a possi- ble way to improve knowledge sharing. We aim to improve knowledge discovery by facilitating connection creation be- tween knowledge sources and users in need of knowledge.
Knowledge sources can be individual content items or other users with similar interests possessing the knowledge. We propose an approach that can suggest 1) content relevant to users and 2) users with similar interest. Below, we present two main steps of our approach.
Step 1. Computing term weights with TF-IDF.
On the first step, we compute the relevance of specific terms (including concepts) for the content items by using a known information retrieval technique, namely term frequency - in- verse document frequency (TF-IDF) as explained in [17].
When computing the weight, TF-IDF considers the frequency of the term inside of a document and its frequency in the whole corpus. In this way, for each content item we obtain a vector that contains weights of individual words or concepts cwcipresent in the content:
cwci=tfci∗idfci, (3) where tfci is the term frequency representing how often the termciappears in the document andidfciis the inverse document frequency indicating how common is the termci in all documents.
Step 2. Scoring relevant items with cosine simi- larity. To obtain for the userusuggested content items or relevant users, we compute the relevance score for the item d using a cosine similarity between the two vectors repre- senting the user and the content:
S(u, d) = V(u)·V(d)
|V(u)||V(d)|, (4) whereV(u) andV(d) are the vectors containing weights
Extracted Text Content Items on platform
Binary Text File .pdf .docx
Image with text .png .jpg .tiff
Image Audio
Video
Content Extraction Plain Text File
Optical Character Recognition Speech-To-
Text
Visual Image Recognition
Visual Video Recognition
Content Analysis
Content and Concepts
Indexing Identified
Concepts Indexed
Identified Concepts and ContentText
Recommender System
Figure 1: A possible pipeline architecture to extract concepts from diverse content types. Dotted lines mark the parts yet to be implemented in Graasp.
Pdf Report
Powerpoint Presentation
Image with Text
Youtube Video
Σw*UA
*DC accessed
rated commented
downloaded Education
Educational psychology Knowledge
Learning
Knowledge Management Human-Computer Interaction Interdisciplinarity
Academia Systems thinking Scientific method Educational technology Virtual learning environment User
Identified Concepts (DC)
Identified User Concepts (UC)
Tracked Activities (UA) Education
Educational psychology Knowledge Learning
Knowledge Management Systems thinking Scientific method Educational technology Virtual learning environment Learning Knowledge Management Human-Computer Interaction Interdisciplinarity Education Educational psychology Academia
Figure 2: A schematic representation of the proposed approach. The system aggregates the concepts from the content as the user interacts with the content.
of the user terms and the document terms computed at Step 1;V(u)·V(d) is a scalar product of the two vectors;|V(u)|
and|V(d)|are Euclidean norms of the vectors.
4. IMPLEMENTATION
To validate the feasibility of the approach and further eval- uate it, we have implemented it in Graasp, a social media platform employed for knowledge management. Graasp sup- ports uploading and storage of content from user devices or the cloud. Graasp provided extraction of text content from multiple file formats, and the activity logging infrastructure was already in place. Still, we needed to extend the plat- form to enable content analytics with concepts extraction, construction of the interests profile, and items recommenda-
tions with Elasticsearch1. Below, we explain the architec- ture of the implemented solution.
4.1 Concept Extraction and Activity Tracking
Concept Extraction. The concepts extraction is done as soon as content is uploaded to Graasp. To extract con- cepts, we have implemented a processing pipeline presented on Figure 1. On the first step, the type of the content is identified, and Graasp tries to extract textual information when possible. For plain text files, it just reads the text content of the file. For binary text files including pdfs and
1Elasticsearch Open Source Engine https://github.com/
elastic/elasticsearch
Microsoft Office formats, we use the textract library2. For images, Graasp tries to perform Optical Character Recog- nition and read the text presented on the image using the tesseract3 library. In the future, we foresee extracting text from Audio and Video files relying on Speech-To-Text tech- nologies (shown with dotted lines on Figure 1) and obtaining concepts for images and videos with the help of visual recog- nition tools [13], for instance using clarifai4. Once the text is available, we analyse its content, identifying the concepts present there. For this purpose, we concatenate the item name, the item description and the extracted content and, at the moment of writing, employ AlchemyAPI5 Concept Tagging to get the concepts. It is worth noting that our approach does not assume a specific concept identification technology, and AlchemyAPI was picked for the reasons of minimal administration and scalability. After the system identifies the concepts, it indexes them in Elasticsearch to- gether with the text content extracted before.
Activity Tracking. Graasp uses ActivityStreams format for capturing user activities on the platform. Some of the actions that the platforms records include access, download, rating, commenting, inviting members and, searching.
4.2 Interests and Recommendations
Constructing Interests Profile. Graasp continuously updates interests profile of the users as they interact with the content. Users interests are displayed next to their pro- file information as demonstrated on Figure 3. The user can adjust her profile by removing individual concepts by press- ing the X button and in this way influence in real-time the content and users suggested by the recommender.
Computing Recommendations. In Graasp, we rely on Elasticsearch for computing recommendations whenever the user wants to see them. Elasticsearch is built on the Lucene6 text search engine that internally employs vector space model, TF-IDF, and cosine similarity when finding relevant items7, similarly as in our proposed approach de- scribed in Section 3.2. We assemble into a single search query all of the concepts from the user interests profile and, whenever present, the terms from the user description as on Figure 3 (1). We run this query against the name, de- scription, content and, concepts fields of the items, assigning different boost weights for matches happening in different fields. The obtained results are presented to the user next to her profile as illustrated on Figure 3.
5. EVALUATION
To understand opinions regarding the approach and its performance when put into practice, we have conducted a preliminary evaluation of the approach implementation in Graasp with pre-service teachers. This section explains in more details the methodology used and the main outcomes.
5.1 Methodology
We have conducted a survey-based preliminary evaluation
2textracthttps://github.com/dbashford/textract
3tesseract Libraryhttps://github.com/tesseract-ocr
4Clarifaihttp://clarif.ai/
5AlchemyAPIhttp://www.alchemyapi.com
6Apache Lucenehttps://lucene.apache.org
7Relevance Scoring https://www.elastic.co/guide/en/
elasticsearch/guide/current/scoring-theory.html
of the developed approach. Surveys are one of the common ways of evaluating recommender systems allowing to col- lect opinions regarding the system from multiple users in a reasonable timeframe [7, 21]. Our goal was to validate if the approach, in general, is useful, if its implementation in Graasp is usable, as well as if the system can identify relevant interests and recommend relevant items.
Participants. We have conducted the survey with six participants of a workshop on inquiry-based learning for pre- service teachers in secondary education. During the work- shop the participants registered in Graasp and carried out on the platform a set of activities during 2 hours. At the end of the session, we asked them to fill in the survey.
Survey Structure. Our survey had three parts8. The first part asked about general disposition towards the in- terests identification and the interests-based recommender.
The second part was the System Usability Scale (SUS) [4]
evaluating the usability of the implemented system. We have selected SUS because of its understood interpretation and robustness [1]. In the third part, we evaluated the quality of the identified interests and recommendations. Two types of questions formed the survey. The first type was questions to indicate the level of agreement with specific statements, where we followed the 5-point Likert scale ranging from 1 - Strongly Disagree to 5 - Strongly Agree to obtain quantita- tive results. The second type was open questions where we asked the responders to provide us with qualitative feedback regarding the approach and its implementation.
5.2 Results
In this section we focus on the main outcomes of the eval- uation. Complete survey results are available online9.
Approach. The users valued positively the idea of us- ing their interests to guide the recommendations and to find other users with similar interests (mean Likert score µ= 3.17 andµ= 3.33 respectively). Besides, the fact of be- ing aware of the inferred interests and the possibility of edit- ing interests were well appreciated (µ= 3.17 andµ= 3.33 respectively). Although the quantitative analysis does not illustrate a high adoption by the users, during the work- shop, the participants were keen on understanding how the interests were extracted and highlighted the novelty of the approach. Further details with the survey results may be found following the URL mentioned above.
Usability. In general, the participants were eager to use the recommender with a certain frequency (µ = 3.33) and did not report major issues regarding complexity, inconsis- tency or difficulty of usage. Just one person considered that she would need technical support or previous background to use the recommender. According to the discussion with this person after the workshop, these answers were partially conditioned by the cognitive load due to the short time avail- able to get used to the platform itself and to integrate all the ideas presented in the workshop. The quantitative results of the SUS questionnaire are also available on-line.
Accuracy. Despite the limited amount of traces collected due to the short time of the user interaction, the results point out that both the interests extracted and the recom- mendations, in general, were relevant (µ= 3.17) and diverse (µ= 3.50). It is noteworthy that when we asked the users to check how many relevant interests and recommendations
8Recommender Evaluation Surveyhttps://goo.gl/Wes6uP
9Evaluation resultshttps://goo.gl/Wes6uP
1 2 3
User Profile with Identified Interests Suggested Content Suggested People
Figure 3: (1) User interests in Graasp as identified by our approach. Suggested content (2) and suggested people (3) based on the user profile information and identified interests.
appeared in the top 10, we discovered two groups. While most of the users reported more that six relevant items, two users got less than two relevant items. We have looked into this case and identified the reason covered below.
Sensitivity to Inaccurate Concepts. In the case when an item that the user interacted with many times has con- cepts identified not accurately, these concepts appear on the top of user interests. We plan to mitigate this problem in the future by introducing a heuristic for not considering con- cepts with low relevance and by limiting the influence of a single item on the overall concept relevancy for the user. Our goal is to make sure that the identified concepts come from many items rather than from many visits to a single item with potentially misidentified concepts. Our expectation is that it will allow reducing the impact of faults in concepts identification on the user resulting user profile.
Privacy Implications. Right now, only the user can see her interests, but we consider putting in place a mechanism that will allow to make validated interests visible to other users of the platform and to make it possible to find the user based on her interests as it was proposed in [15]. However, based on the evaluation, while some of the participants were eager to make their interests visible, others were reluctant.
Thus, it will be necessary to allow users configure the visibil- ity of their interests to preserve their privacy, following the recommendations provided in codes of practice for learning analytics [20].
6. CONCLUSIONS AND FUTURE WORK
In this paper we proposed a new approach to building user interests profile based on 1) content analytics providing the system with the concepts present in the content and 2) ac- tivity tracking allowing the system to know how the user interacted with the content. We have used the extracted in- terest concepts to recommend relevant content and people.
Further on, we have implemented the proposed approach in Graasp, a knowledge management system. Graasp was used in a workshop to support teachers when building inquiry learning spaces for their students. Thus, we have evalu- ated the approach with the teachers, and the evaluation has demonstrated that the proposed approach can identify rel- evant user interests and recommend relevant content based
on the identified interests. At the same time, the evalua- tion has unveiled sensitivity of the approach to inaccurately identified concepts that we plan to overcome in the future.
While we draw our experience and motivation from the edu- cational context, our contributions have a broad impact and can be applied for content repositories, where it is possible to obtain content analytics and track activities performed by the users (e.g., Google Drive and Dropbox).
Looking Outside. In this study, we analyzed the con- tent and recorded the activities limited to the scope of the content repository. However, in the current technological landscape, the interactions are getting more distributed of- ten spanning across multiple platforms. Studies suggest that combining data obtained from several platforms could allow creating a more accurate user interests profile [8]. In the fu- ture, we plan to extend the architecture of Graasp to aggre- gate the content and the interactions outside of the system.
Incorporating Relevance Scores. At the moment, when computing the similarity score for the recommended items we consider the fact of the concept presence but do not take into account the available concept relevance scores.
Incorporating the relevance scores available for user interest concepts and content concepts when computing the user- content relevance score may lead to more relevant recom- mendations since it will promote the results with similar highly relevant concepts.
Recommender Adaptability. One potential downside of our approach could be related to its limited ability to react timely to change in the user interests reflected in her interac- tions. This happens since the concepts the user accumulated at some point through her interaction history maintain the same score indefinitely. One of the possible solutions to this problem is to introduce the forgetting function as suggested in [22], so that as time goes the concepts that are not en- countered anymore get their relevance score reduced.
Substantial Evaluation. This paper presented a pre- liminary evaluation of the recommender to provide early feedback. We are planning to conduct an evaluation with more users that used Graasp for longer periods of time so that more activity traces are available. We also expect these users to have established expectations regarding their inter- ests when interacting with the platform.
7. ACKNOWLEDGMENTS
This research is partially funded by the European Union in the context of the Go-Lab project (Grant Agreement no.
317601) under the Information and Communication Tech- nologies (ICT) theme of the 7th Framework Programme for R&D (FP7). This document does not represent the opin- ion of the European Union, and the European Union is not responsible for any use that might be made of its content.
8. REFERENCES
[1] A. Bangor, P. T. Kortum, and J. T. Miller. An empirical evaluation of the system usability scale.Int.
J. Hum. Comput. Interact., 24(6):574–594, 2008.
[2] E. Bogdanov, F. Limpens, N. Li, S. El Helou, C. Salzmann, and D. Gillet. A social media platform in higher education. InProceedings of the Global Engineering Education Conference, EDUCON, pages 1–8, Apr. 2012.
[3] I. Bosni´c, K. Verbert, and E. Duval. Automatic keywords extraction - a basis for content recommendation. InProceedings of the 4th
International Workshop on Search and Exchange of e-learning Materials, pages 51–60, Sept. 2010.
[4] J. Brooke. SUS: A quick and dirty usability scale. In P. W. Jordan, B. Weerdmeester, A. Thomas, and I. L.
Mclelland, editors,Usability evaluation in industry, pages 189–194. Taylor and Francis, 1996.
[5] H. Drachsler, K. Verbert, O. C. Santos, and
N. Manouselis. Panorama of recommender systems to support learning. InRecommender Systems Handbook, pages 421–451. Springer US, 2015.
[6] S. El Helou, C. Salzmann, and D. Gillet. The 3A personalized, contextual and relation-based recommender system.J. Univers. Comput. Sci., 16(16):2179–2195, 2010.
[7] M. Erdt, A. Fernandez, and C. Rensing. Evaluating recommender systems for technology enhanced learning: A quantitative survey.IEEE Trans. Learn.
Technol., 8(4):326–344, Oct. 2015.
[8] I. Guy, U. Avraham, D. Carmel, S. Ur, M. Jacovi, and I. Ronen. Mining expertise and interests from social media. InProceedings of the 22nd International Conference on World Wide Web, WWW, pages 515–526, New York, NY, USA, 2013. ACM.
[9] H. Harkous, R. Rahman, B. Karlas, and K. Aberer.
The Curious Case of the PDF Converter that Likes Mozart: Dissecting and Mitigating the Privacy Risk of Personal Cloud Apps. InProceedings of the 16th Privacy Enhancing Technologies Symposium, PETS, Darmstadt, Germany, 2016.
[10] J. Kim, S.-W. Li, C. J. Cai, K. Z. Gajos, and R. C.
Miller. Leveraging video interaction data and content analysis to improve video learning. InProceedings of the CHI 2014 Learning Innovation at Scale workshop, 2014.
[11] K. Kitto, S. Cross, Z. Waters, and M. Lupton.
Learning analytics beyond the LMS: the connected learning analytics toolkit. InProceedings of the 5th International Conference on Learning Analytics And Knowledge, LAK, pages 11–15, New York, NY, USA, March 2015. ACM.
[12] V. Kovanovi´c, S. Joksimovi´c, D. Gaˇsevi´c, M. Hatala, and G. Siemens. Content analytics: the definition, scope, and an overview of published research.
Handbook of Learning Analyitcs, 2015.
[13] A. Krizhevsky, I. Sutskever, and G. E. Hinton.
ImageNet classification with deep convolutional neural networks. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, editors,Advances in Neural Information Processing Systems 25, pages 1097–1105.
Curran Associates, Inc., 2012.
[14] M. G. Moore. Editorial: Three types of interaction.
The American Journal of Distance Education, 3(2):1–6, 1989.
[15] W. Pohs, G. Pinder, C. Dougherty, and M. White.
The lotus knowledge discovery system: Tools and experiences.IBM Syst. J., 40(4):956–966, 2001.
[16] Ralf Klamma, Mohamed Amine Chatti, Erik Duval, Hans Hummel, Ebba Thora Hvannberg, Milos Kravcik, Effie Law, Ambj¨orn Naeve, and Peter Scott.
Social software for life-long learning.Journal of Educational Technology & Society, 10(3):72–83, 2007.
[17] J. Ramos. Using tf-idf to determine word relevance in document queries. InProceedings of the first
instructional conference on machine learning, pages 1–4. cs.rutgers.edu, 2003.
[18] M. J. Rodr´ıguez-Triana, L. P. Prieto, A. Vozniuk, M. Shirvani Boroujeni, B. A. Schwendimann, A. C.
Holzer, and D. Gillet. Monitoring, Awareness and Reflection in Blended Technology Enhanced Learning:
a Systematic Review.International Journal of Technology Enhanced Learning, In press.
[19] J. L. Santos, K. Verbert, J. Klerkx, E. Duval, S. Charleer, and S. Ternier. Tracking Data in Open Learning Environments.J. Univers. Comput. Sci., 21(7):976–996, 2015.
[20] N. Sclater. Code of practice for learning analytics. a literature review of the ethical and legal issues.
Technical report, JISC, 2014.
[21] G. Shani and A. Gunawardana. Evaluating
recommendation systems. InRecommender Systems Handbook, pages 257–297. Springer US, 2011.
[22] H. Th¨us, M. A. Chatti, R. Brandt, and U. Schroeder.
Evolution of interests in the learning context data model. InDesign for Teaching and Learning in a Networked World, pages 479–484. Springer, 2015.
[23] A. Vozniuk, S. Govaerts, and D. Gillet. Towards portable learning analytics dashboards. InProceedings of the 13th International Conference on Advanced Learning Technologies, ICALT, pages 412–416, Beijing, China, July 2013. IEEE.
[24] A. Vozniuk, A. Holzer, S. Govaerts, J. Mazuze, and D. Gillet. Graspeo: a social media platform for knowledge management in NGOs. InProceedings of the 7th International Conference on Information and Communication Technologies and Development, page 63, Singapore, Singapore, 15 May 2015. ACM.
[25] V. A. R. Zaldivar, R. M. Crespo Garc´ıa, B. Daniel, P. Abelardo, and others. Automatic discovery of complementary learning resources. InProceedings of the 6th European Conference of Technology Enhanced Learning, EC-TEL, pages 327–340, Palermo, Italy, 2011.
