A Learning Algorithm for Episodes

HAL Id: hal-01847561

https://hal.laas.fr/hal-01847561

Submitted on 23 Jul 2018

HAL

is a multi-disciplinary open access archive for the deposit and dissemination of sci- entific research documents, whether they are pub- lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire

destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

A Learning Algorithm for Episodes

Tom Obry, Audine Subias, Louise Travé-Massuyès

To cite this version:

Tom Obry, Audine Subias, Louise Travé-Massuyès. A Learning Algorithm for Episodes. 28th Inter-

national Workshop on Principles of Diagnosis (DX 2017), Sep 2017, Brescia, Italy. 5p. �hal-01847561�

A Learning Algorithm for Episodes

Tom Obry

and Audine Subias

and Louise Travé-Massuyès

1

LAAS-CNRS, Université de Toulouse, CNRS, INSA, Toulouse, France

2

ACTIA, 5 Rue Jorge Semprun, 31432 Toulouse

Abstract

Sequences of events describing the behavior and actions of agents or systems can be collected in several domains. An episode is a collection of events that occur in a given partial order. By performing a recognition of recurrent episodes in several sequences and comparing them, it is pos- sible to determine a pattern common to all the se- quences. In this paper, we propose an approach to recognize episodes that are common in a set of event sequences. The method described is applied to the automotive domain for learning diagnosis procedures.

1 Introduction

In the automotive domain, as vehicles become increasingly complex, it becomes difficult to efficiently train garage me- chanics in front of the ever growing technology. Faults become more and more difficult to diagnose and training courses multiply to manage new failures. The purpose of this work is to improve the daily work of garage mechan- ics by providing them with assistance in the diagnosis and repair tasks.

A diagnosis session corresponds to a set of tests and actions performed by the mechanics via a diagnostic tool performed on a vehicle with a breakdown. Each diagno- sis session performed is saved in the form of a trace. The goal is to compare all of these diagnosis traces correspond- ing to the same symptoms to extract the common actions.

The aim of this work is to determine the most common ac- tion plan that successfully diagnoses a breakdown or that carries out a repair. We propose an approach based dis- covery and learning patterns [1] [2]. It is inspired by [3]

[4] that proposes an algorithm to learn chronicles. Chron- icles are temporal patterns based on a set of events and temporal constraints relating their occurence dates. They are used to represent the dynamic behavior of complex systems in an event-based fashion and have found several applications in the supervision and diagnostic field [5; 3;

6]. In our case, the occurence dates are not recorded in the traces that are given as input sequences for the learning algo- rithm. The reason is that actions performed by the mechan- ics are not constrained over time. This is why we turned to a particular type of chronicles called "episodes" [7] [8].

Episodes just specify the precedence order of events.

This paper is organized as followed. Section 2 introduces the notion of episode. In the section 3, a new algorithm to

learn episodes (LAE) is described. Section 4 presents the test and results. Section 5 presents a standard algorithm for episode learning and a comparison with the new algo- rithm. Finally, section 6 concludes with the perspectives of this work.¹

2 Basic concepts

We consider the input as a sequence of events, where each event has an associated date of occurrence. Given a setE ofevent types, aneventis a pair (e_i, t_i), wheree_i ∈Eis an event type andtis an integer, the (occurrence) dateof the event.

Definition 1. Anevent sequence sonEis a triple (s,T_s, Te)

s=<(e1, t1),(e2, t2), ...,(en, tn)> (1) is an ordered sequence of events such that e_i ∈ E for all i= 1, ..., n, andti ≤ti+1for alli= 1, ..., n−1. Further on,T_sandT_eare integers:T_sis called the starting time and Tethe ending time, andTs ≤ ti ≤ Tefor alli = 1, ..., n [7].

Episodes are defined as a collection of events that occur frequently in a sequence. An episode is a set of partially or totally ordered event types [7].

Definition 2. An episodeψ= (V,≤, g)is a set of nodesV, an ordered partial≤onV and a mappingg:V →Ewhich associates a node with an event type.

The interpretation of an episode means that the events of g(V)must appear in the order described by≤.

The difference with chronicles is at the level of the temporal bounds: the intervals in episodes are not bounded. This makes it possible to consider only the notion of precedence between the events:tj−ti∈[0,+∞]ortj−ti∈[−∞,0].

Definition 3. An episodeβ= (V⁰,≤⁰, g⁰)is a sub-episode ofα= (V,≤, g), denotedβ ≤αif there exists an injective mappingf:V⁰ →V such thatg⁰(v) =g(f(v))for allv∈ V⁰ and for allv,w∈V⁰ withv≤⁰ walsof(v)≤g(w).

An episodeαis a super-episode ofβif and only ifβ ≤ α. There are two classes of episodes: serial episodes and parallel episodes.

1This work has been supported by the company ACTIA, one of the market leaders for diagnostic tools in the automobile domain.

Definition 4. An episodeαis serial if the relation≤is a total order (i.e.,x≤y ory ≤xfor allx,y ∈V). If the partial order≤is trivial (i.e.,xy for allx,y∈V such thatx6=y), the episode is parallel [9].

Figure 1: Example of a serial episode and a parallel episode Figure 1 shows two examples of episodes. The episode on the left is a serial episode because the order is total between A,B andC. The other episode is parallel. Indeed, there is no constraint on the partial order on the event typesA, BorC[7]. Episodes issued from the diagnostic tool traces are parallel episodes because the order between the event types is not total. This means that the order of the actions for diagnose a vehicle is not total.

3 Learning Algorithm for Episodes (LAE)

We present in this section, a new algorithm which can compute a set of parallel episodes from a set of input sequences. The solution includes events that are common to all input sequences. We detail the different steps that make up the algorithm. For a better comprehension of the steps, we use 3 sequences,s1 ∈S,s2 ∈Sands3 ∈S(whereS is the set of the input sequences):

s1=<(A,1),(B,2),(C,3),(D,4),(A,5)> (2) s2=<(A,1),(B,2),(C,3),(D,4),(E,5),(A,6)> (3) s3=<(B,1),(C,2),(A,3),(D,4),(A,5),(E,6)> (4) The sequences are stored in a table. Each cell contains 2 other cells, where the first contains the event types and the second, the dates of occurrence.

3.1 Searching for common events

The first step consists in sorting the events that are common and those that are not. Common events are searched in the input sequences then, sequences of entries are modified to keep only the common events. In this context, events are different screens used during the diagnosis session.

Experienced mechanics receive training on how to diagnose a vehicle. They know how to use the diagnosis tool and which screens to navigate. The events that are not in all input sequences are considered as beginner mechanic’s actions.

A B C D

Table 1: ListComof common events ins1, s2ets3

Table 1 represents all the common event types located inS.

Here, the common event types areA,B,C andD. Since eventEis not present in all the sequences, it is not present in the listCom.

With these common events, events that are not in the ini- tial sequences are removed. Thek^theventekof thei^thse- quences_iis compared with thej^thevente_jof the collection of common eventsCom[j]. When the two events compared are identical, the algorithm indexes the event in the first cell and the date of occurrence in the second cell onsi.

s1 s2 s3

ABCDA 12345 ABCDA 12346 BCADA 12345 Table 2:s1, s2ets3with only common events Table 2 summarizes, for each sequence, the event types that are common in all sequences and their occurrence positions in the sequence. All the events located ins₁are inCom[1]

because they are present in all the sequences. The event type E is not present in thes2ands3because E is not present in all the sequences.

3.2 Computing the number of occurrences of an event type innsequences and computing the pairs of events

In this step, the number of occurrences of each event type is found in then input sequences. These information are stored in a table calledSet. It is composed ofkcolumns andirows, wherekcorresponds to the event type and i, the input sequence number. Each cell is composed of 2 more cells. The first contains the frequency ofeklocated in thei^th sequence. The second cell contains a vector of "0" and "1".

"1" indicates the position of the eventekin the sequencei.

A B C D

2 [100010] 1 [010000] 1 [001000] 1 [000100]

2 [100001] 1 [010000] 1 [001000] 1 [000100]

2 [001010] 1 [100000] 1 [010000] 1 [000100]

Table 3: Synthesis of the information inSet Table 3 shows the event types common inSet, their occur- rence positions for each sequence and the position of each event in each sequence.

To determine the maximal frequency of each event, noted F req(ek), the minimal number of occurrences of each event, notedmin(Occ(ek)), for each sequencesis required.

F req(e_k) =min(Occ(e_k)∈s, s∈S,∀s) (5) Each frequency is stored in a tableF req, under the format (ek,F req(ek)).

A B C D

2 1 1 1

Table 4: Maximal frequencies in the table F req of each event types ins₁,s₂,s₃

In the table 4, we can see the maximal frequency of each common event type. In this example, this information equals 2 for the event Aand 1 for the others event types.

To calculate the pairs available in each sequence, all combinaisons ofF req[i] withF req[j] must be performed.

AB AC AD BC BD CD

2 2 2 1 1 1

Table 5: Set of pairs available in sequencess1,s2,s3

All the pairs and their maximal frequencies in S are pre- sented in the table 5. In this case, the maximal frequency for pairs(A, B),(A, C)and(A, D)is 2 and for pairs(B, C), (B, D)and(C, D), the value of the maximal frequency is 1.

3.3 Determination of the direction of the pairs We are going to explain, for each pair of event types, a re- lation of unique precedence, that is to say present on the ninput sequences. In other words, it is a matter of deter- mining thedirection of a pair of events. Let’s take our 3 examples:

s1=<(A,1),(B,2),(C,3),(D,4),(A,5)> (6) s₂=<(A,1),(B,2),(C,3),(D,4),(E,5),(A,6)> (7) s₃=<(B,1),(C,2),(A,3),(D,4),(A,5),(E,6)> (8) For the pair(A, B),Ais before and afterB ins1ands2. The label "↔" is allocated to the pair (A,B) to indicate that the pair(A, B)can occur in any order, i.e. Acan precede BorBcan precedeA. For example, the pair A↔B means it is possible to go to the event B from A and return to the event A. Pairs(B, C),(B, D)and(C, D)keep always the same direction in all sequences. The label is given for those two pairs of events "→" becauseCalways followsB,Dis always afterBandDalways occurs afterC. To summarize, the label "→" can only be assigned ifAis followed byBin all the sequences (conversely, "←" forB is followed byA in the all the sequences). The label "↔" is assigned to a pair:

• if there is a label "→" and a label "←" in at least one sequence. It indicates that both directions are possible.

• if the two directions are possible in the same sequence.

The tableSetis organized by events in columns indexedk.

Every eventekgives rise to a macro-column with two sub- columns indexedk₁andk₂. The lines indexedicorrespond to the input sequences. The vectors of "0" and "1", located in the tableSetare studied because they contain the occur- rence positions of the events. Set(i, k1)provides the num- ber of occurence of the event of columnkin the sequencei andSet(i, k2)is the vector from which we can retrieve the event position in the sequence. From this information, we can obtain the direction of every pair in every sequence. To save the direction of the pairs, the variableInteris defined and can take several values:

• Inter_i(k, l)=+1 ife_lfollowse_kin thei^thsequence,

• Inter_i(k, l)=-1 ife_kfollowse_lin thei^thsequence,

• Inter_i(k, l)= 0 ife_kprecedes and followse_lin thei^th sequence.

To find the direction of the pairs in the n sequences, we proceed as follows:

• ifPn

i=1Interi(k, l) =n, the label assigned to the pair (e_k, e_l)is "→",

• ifPn

i=1Inter_i(k, l) = −n, the label assigned to the pair(ek, el)is "→",

• otherwise, the label assigned to the pair(ek, el)is "↔".

3.4 Test of consecutivity

We aim to reduce the set of solutions by adding the notion of consecutiveness. If two events are always consecutive on thensequences (i.e there is no other event between them) then we can say that the pair is consecutive. In this case, a "No_Event" constraint between the two events is set [4].

The vectors of the occurrence positions Set(i, k2) of the first pair eventsek andelare studied. When the first "1" of theekvector is detected, the value of the following column of the vector of the eventelis studied: if it is a "1", then the pair studied is consecutive at least once in one sequence and the algorithm continues the test with the other sequences and with the others pairs. When all pairs are calculated, they are assembled to be interpreted with a contraints graph. A con- traints graph represents the result with a graphic way. Each node represents an event and the index next to the event in- dicates the maximal frequency of each event. The arc cor- responds of the no bounded time contraints. A dashed line represent the specific relation "No_Event" between a pair of events. Arrows on the arcs model the direction of each pair.

Figure 2: Constraints graph of the 3 test sequences Figure 2 represents the constraints graph of the 3 tests sequences. We can remark that we have 2 occurrences of the event typeA and 1 occurence for the event types B, CandD, 3 pairs with a double direction ((A, B),(A, C), (A, D)), 3 pairs with an unique direction ((B, C),(B, D), (C, D)). The pair(B, C)have an dashed arrow to represent the label "No_Event" because it is consecutive on the 3 input sequences.

4 Tests and results

We applied the LAE algorithm with the ACTIA’s traces. The actual traces were not available during the tests, they were simulated on a diagnostic tool. The diagnostic tool allows the mechanic to carry out his tests and measurements on the vehicle in real time. The mechanic selects actions displayed on the screen. The tool works by changing screens. The users can interact via the tool to test the various functions of the vehicle.

Figure 3: Overview of the context

The figure 3 explains the architecture of the context. The mechanic tests the vehicle using the diagnostic tool. The tool provides a trace summing all the actions the garage owner did in the diagnosis session. The LAE algorithm uses all the diagnosis traces corresponding to common symp- toms. They are compared to provide the common approach to all entries in the form of parallel episodes. For a good precision, we went into 4 different garages to ask owners the procedures to follow to diagnose a failure with a com- mon symptom.

Figure 4: Extract of an Actia’s sequence

The extract from figure 4 represents the beginning of a simulated trace on the diagnostic tool. The first accessible screen offers a choice on the brand of the vehicle to be di- agnosed, "Peugeot" or "Citröen". This example was car- ried out on a vehicle "Citröen" and the model was a "C4 Picasso". The first line corresponds to the selection of the

"Citröen" button of the diagnostic tool. A change of screen displays a list of vehicles of the mark to be selected. The second line indicates that the mechanic was on the "Cit- röen" vehicle selection menu and the third line shows he se- lected the "C4PB78" button, which corresponds to the "C4 Picasso" model, etc. The simulated sequences have an aver- age of 65 events and there are 4. The result consists of 78 pairs including 13 elements in common. Note that in this case of study, two actions (i.e. events) without preference of order cannot be done sequentially in one or the other order equivalently. The generated result should be filtered to take this feature into account. The algorithm is able to consume traces of ACTIA and generate a satisfactory result from a temporal and qualitative point of view. The execution time is 0.7 seconds with ACTIA traces. The tests were carried out on an ASUS PC, under windows 7 with an Inter Core i7 processor - 2670QM, 2.2GHz, 6GB of RAM. The work carried out is considered as a proof of concept. Neverthe- less, this work need to be continued by a deeper analysis of the results. Indeed, the constraints graph represents the raw episode of theninput sequences. As the result was not con- fronted with the mechanics, the pertinence of the episode was not evaluated.

5 A standard algorithm for episode learning

In this section, we present an existing algorithm that allows learning episodes innsequences [7]. This algorithm allows to extract the different serial episodes and parallel episodes that are located in a sequences. The sequences provided by the diagnostic tool being parallel, we only present the part dealing with this type of episodes.

The algorithm recognizes parallel episodes in a sequences.

This step creates the database that contains all the episodes of thessequence. The algorithm uses two windowsw = (w, ts, ts+win)etw⁰ = (w⁰, ts+ 1, ts+win+ 1)where wandw⁰ are the sequences contained mutually inw and w⁰,tsis the starting time of the window andwinis the di- mension of a window such thatwidth(w) =win[7]. The sequences contained in w etw⁰ will therefore be similar.

After the recognition of the episode inw, an update of the data structure is performed to get the window to move to the w⁰ episode in the same sequence s. The recognition of a candidate parallel episode α is done with a counter α.event_count. This counter indicates how much events of αare present in the window. Whenα.event_countis equal to the length ofα, the totality ofαis present in the window. α.f req_countis incremented by 1. That counter represents the number of windows whereαis contained en- tirely. Candidates episodes are indexed by the number of events of each type they contain. All episodes that contain atype A events will be in thecontain(A,a).

An episodeαis stored in a table of events sorted alpha- betically. Each event ofαis indexed between brackets. For example, an episodeβwith events types B, D, D, E is rep- resented by a tableβwithβ[1] =B,β[2] =D,β[3] =D, β[4] =E. Collections of episodes are sorted alphabetically in a tableF, where thei^thepisode of the collection is rep- resented byF[i]. All the episodes that share the first event type are consecutive in the episode collection. A maximum sequence of consecutive episodes of sizel, sharing thel−1 first events is called a block. Candidate episodes are iden- tified by creating all possible combinations of two episodes of the same block [7].

The collection of frequent episodes is a list featuring episodes that most frequently appear in the n sequences.

The algorithm computes the collection of frequent episodes that make up the sequence s from the class of paral- lel episodes. The search is done by levels of length l on episodesαin the episode class following the sub-episode re- lationship. The search begins with the most general episode (one containing only one event). At each level, the algo- rithm calculates the collection of episodes and their frequen- cies.

The frequency of an episode is defined as the ration be- tween the number of the window where the episodeαap- pear entirely and the total window number. With a sequence of eventss, a window of dimensionwin, the frequency of an episodeαinsis:

f r(α, s, win) =|{w∈W(s, win)|α∈w}|

|W(s, win)| (9) where s is the number of windows is the α episode is fully included, W is the set of all windows w ons. For a defined threshold frequency thres_fr, α is frequent if

f r(α, s, win)≥min_f r.

A rule for predicting frequent episodes that respects,win, etmin_f r is denoted F(s, win, min_f r). A prediction rule is aβ ⇒γexpression whereβandγare episodes such as β ≤ γ. The ratio f r(γ,s,win)

f r(β,s,win) is called the confidence of the episode prediction rule. This confidence can be inter- preted as a conditional probability that the entireγis present in the window, with a givenβ.

5.2 Comparaison of LAE and the standard algorithm

In this section, we compare the algorithm LAE presented in section 3 to the standard algorithm of [7] presented in sec- tion 5.1. The first step of the standard algorithm is to find all candidate episodes of thensequences. Once all the episodes are found, they are stored and classed alphabetically in order to start creating all possible combinations of two episodes of the same block. Once the candidate episodes are gen- erated, they are stored in theF(s, win, min_f r)episode collection. The prediction and confidence rules are then de- duced from the previous result. The LAE algorithm pro- ceeds quite differently: it begins by looking for events that are not in common in thensequences and then it removes them from the sequences to keep only the common events. It determines the maximum occurrence number of each event, which allows us to easily find the maximum frequency of each pair of events on thensequences.

The difference with the standard algorithm is that it finds the episodes in the form of a constraint graph describing all the possible episodes. The standard algorithm makes it pos- sible to obtain an episode table directly.

The algorithms have their differences, however: with the F(s, win, min_f r)event collection, it is possible to pre- dict or anticipate the behavior of the sequence in its integrity.

The algorithm thus provides a set of episodes as well as their prediction rules.

On the other hand, the LAE algorithm provides important information thanks to the consecutive test. It makes it pos- sible to express the fact that there can be no intermediate events between the two events of a pair on thensequences of inputs. An intermediate event is an event of a different type than those defined in the pair.

The comparison with the Mannila algorithm will be pursued when we’ll have a larger set of data. Then, real tests and fur- ther analysis could be done.

6 Toward the Ph.D work

The thesis is the continuation of the presented work. The Ph.D is a collaboration with the ACTIA company which de- veloped a software named ACTI-DIAG destined to mechan- ics. It allows to dialogue with the set of calculators (airbag, injectors, ...) of a vehicle and to pick up all the available in- formation inside the car. These information can then be ex- ploited in conjunction with information from the manufac- turer or independent actors to finally allow the mechanic to locate the component to be replaced. In the field of automo- tive diagnostics, the number of identical vehicles circulat- ing makes it possible to capitalize a great amount of knowl- edge. This capitalization then makes it possible to repair a vehicle more quickly by taking advantage of the experience gained on similar vehicles having already encountered the same problem. Currently, the information collected on the vehicle are:

• The type of the vehicle.

• The set of calculators et their actual software version.

• The set of default codes (also called Data Trouble Code) on calculators with the occurrence time.

• The context (values of parameters judged relevant).

• etc...

The objectives of the thesis is to analyze (semi-) automati- cally this set of data to reduce the updating time of the di- agnosis and repair methods by the manufacturer, but also to reduce the time of the diagnosis and repair stages when a vehicle is in a workshop. This reduction can be addressed by:

• An update of existant knowledges (the list of possible causes for a given symptom, the probability of each fault occurrences about a vehicle, ...).

• A ranking of knowledges proposed to the mechanic (most probable hypothesis related to a symptom, the

"incident" files having most often completed, ...).

• The creation of a new incident file about the most re- current fails in the network.

• Optimize existing diagnosis procedures to minimize their journey time by taking into account the time re- quired to complete each of their tests and the probabil- ity of occurrence of each fault.

• The learning of new diagnosis procedures by using col- lected data (actual DTCs, the context of the raise of DTCs, ...) in order to distinguish different possible faults for a or several given symptoms.

ACTIA is setting a network of 50 workshops for collecting data from different diagnosis with the symptoms and causes labels. This operation will allow to do real tests and com- parisons with standard algorithms. The deployment of this network is planned for the end of November. Until that time, the study is performed without the label information. In first step, for a given ECU (Electronic Control Unit), the goal is to find co-occurrence of DTCs on vehicles. As DTC are discrete data (for example, P318D is the DTC for a com- munication error for the Nissan Leaf model), learning al- gorithms like clustering with non-ordered discrete data [10]

andk-Nearest Neighbor Searching in non-ordered discrete data space [11] are investigated.

7 Conclusion and perspectives

The work presented in this paper is part of a learning process for generic automotive diagnostics. The objective is to auto- matically generate a model of the diagnosis procedures used by mechanics to diagnose a given fault. This is performed from a set of action sequences carried out during several diagnosis sessions referring to the fault. Since the traces recorded by the diagnostic tool of our case study are time- less, we have directed ourselves to the development of our own learning algorithm: Learning Algorithm for Episodes.

This algorithm allows to learn temporal patterns where time information is captured only through the notion of prece- dence. The tests carried out made it possible to validate the first version of the algorithm presented in the paper.

This work has reinforced the interest of automatic learn- ing to exploit the feedback from the diagnosis sessions. Sev- eral perspectives have been identified. First, it is necessary

to test the algorithm with a larger volume of input data and to evaluate its efficiency. Also, the consecutive test need consolidation. From an application point of view, the idea is to use the results of LAE to improve the training tools of garage mechanics. As the complexity of vehicles increases, paper is not suffisant anymore. Our industrial partner (AC- TIA) develops a Serious Game to facilitate the learning of new methods of diagnosis and repair. A Serious Game is a software that combines a serious intention with playful springs. Episodes provided by LAE can be turned into train- ing scenarios for the Serious Game for the diagnosis and re- pair tasks.

The problem of knowledge capitalization addressed in this work is a real challenge for ACTIA company. The collab- oration with ACTIA on this research area is going in the context of a Ph.D.

Acknowledgements

We would like to thank our contacts in the company ACTIA for their valuable comments and recommendations.

References

[1] R. Agrawal and R. Srikant. Fast algorithms for min- ing association rules. Proc. 20th Int. Conf. on Very Large Data Bases, Santiago, Chile, pages 487–499, Jan 1994.

[2] Mitsa and Theophano. Temporal data mining. CRC Press, 2010.

[3] A. Subias, L. Travé-Massuyès, and E. Le Corronc.

Learning chronicles signing multiple scenario in- stances. InThe 19th World Congress - The Interna- tional Federation of Automatic Control, Cape Town, South Africa, August, 24-29 2014.

[4] Discovering chronicles with numerical time con- straints from alarm logs for monitoring dynamic sys- tems, July 21-26 1999.

[5] C. Dousson, P. Goborit, and M. Ghallab. Situation recognition : representation and algorithms. In IJ- CAI : International Joint Conference on Artificial In- telligence, pages 166–172, Chambéry, France, August 1993.

[6] D. Cram, B. Mathern, and A. Mile. A complete chron- icles discovery approach : application to activity anal- ysis. Expert Systems, 29(4):321–346, 2012.

[7] A.Inkeri Verkalo, H. Mannila, and H. Toivonen.

Discovery of frequent episodes in event sequences.

Data Mining and Knowledge Discovery, 1(3):259–

289, September 1997.

[8] K. Amphawan, J. Soulas, and P. Lenca. Mining top-k regular episodes from sensor streams. In7th Interna- tional Conference on Advances in Information Tech- nology, pages 76–85, 2015.

[9] H. Mannila and H. Toivonen. Discovering generalized episodes using minimal occurrences. InKDD-96 Pro- ceedings, 1996.

[10] A. Watve, S. Pramanik, S. Jung, B. Jo, S. Kumar, and S. Sural. Clustering non-ordered discrete data. Jour- nal of Information Science And Engineering, 30:1–23, 2014.

[11] D. Kolbe, Q. Zhu, and S. Pramanik. On k-nearest neighbor searching in non-ordered discret data spaces.

InIEEE 23rd International Conference of Data Engi- neering, Istanbul, Turkey, April 2007. IEEE.