6.4 Measuring the Efficiency of the Cascade Classification Scheme
The efficiency of the adapted cascade classification scheme was measured in terms of theMean Absolute Errorand theRanked Scoringmeasures. The Mean Absolute Error (MAE) constitutes the most commonly measure used to evaluate the efficiency of RS. More formally, the MAE concerning useruat foldkmay be defined as:
MAE(u,k)= 1
|P(u,k)| + |N(u,k)|
v∈P(u,k)∪N(u,k)
|Ru,k(v)−Ru,k(v)| (6.14)
The Ranked Scoring (RSC) [1] assumes that the recommendation is presented to the user as a list of items ranked by their predicted ratings. Specifically, RSC assesses the expected utility of a ranked list of items by multiplying the utility of an item for the user by the probability that the item will be viewed by the user. The utility of an item is computed as the difference between its observed rating and the default or neutral ratingdin the domain, which can be either the midpoint of the rating scale or the average rating in the dataset. On the other hand, the probability of viewing decays exponentially as the rank of items increases. Formally, the RSC of a ranked list of itemsvj ∈P(ui,k)∪N(ui,k)sorted according to the indexjin order of declining Rui,k(vj)for a particular useruiat foldkis given by:
RSui,k=
vj∈P(ui,k)∪N(ui,k)
max{Rui,k(vj)−d),0} × 1
2(j−1)(i−1) (6.15) Having in mind that the set of testing patterns for the first-level classifier at foldk is formed by the patterns pertaining to the setsC0(u),C1(u,k),C2(u,k)andC3(u,k), we may write that
|P(u,k)| = |C1(u,k)| + |C2(u,k)| + |C3(u,k)| (6.16) and
|N(u,k)| = |C0(u)|. (6.17) According to Eqs.6.16 and6.17, we may define thetrue positive rate (TPR), false negative rate(FNR),true negative rate(TNR), andfalse positive rate(FPR) concerning userufor thekth fold of the testing stage as follows:
TPR(u,k)= TP(u,k)
|P(u,k)| (6.18)
FNR(u,k)= FP(u,k)
|P(u,k)| (6.19)
TNR(u,k)= TN(u,k)
|N(u,k)| (6.20)
and
FPR(u,k)= FP(u,k)
|N(u,k)| (6.21)
It is important to note that the quantities defined in Eqs.6.18,6.19,6.20, and6.21 refer to the classification performance of the first-level classifier in the adapted cas-cade classification scheme. More specifically, True Positive,TP(u,k), is the number of positive/desirable patterns that were correctly assigned to the positive class of patterns while False Negative,TN(u,k), is the number of positive/desirable patterns that were incorrectly assigned to the negative class of patterns. Similarly, True Neg-ative,TN(u,k), is the number of negative/non-desirable patterns that were correctly assigned to the negative class of patterns, while False Positive,FP(u,k), is the num-ber of negative/non-desirable patterns that were incorrectly assigned to the positive class. More formally, having in mind Eq.6.13, the above quantities may be described as follows:
TP(u,k)= {v∈P(u,k):fu,k(v)= +1} (6.22) FP(u,k)= {v∈N(u,k):fu,k(v)= +1} (6.23) TN(u,k)= {v∈N(u,k):fu,k(v)= −1} (6.24) FN(u,k)= {v∈P(u,k):fu,k(v)= −1} (6.25) Computing the mean value for the above quantities over different folds results in the following equations:
TPR(u)= 1 K f∈F
TPR(u,k) (6.26)
FNR(u)= 1 K f∈F
FNR(u,k) (6.27)
TNR(u)= 1 K f∈F
TNR(u,k) (6.28)
FPR(u)= 1 K f∈F
FPR(u,k) (6.29)
It is possible to bound the MAE for the complete two-level classifier according to its performance during the second stage of the multi-class classification scheme.
6.4 Measuring the Efficiency of the Cascade Classification Scheme 109 The best case scenario concerning the classification performance of the second-level (multi-class) classifier suggests that all the true positive patterns, which are passed to the second classification level, are correctly classified. Moreover, the best case scenario requires that all the false negative patterns of the first classification level originated fromC1. Thus, the following inequality holds:
∀u∈U∀k∈ [K]MAE(u,k)≥ FN(u,k)+FP(u,k)
|P(u,k)| + |N(u,k)| (6.30) Given Eqs.6.18,6.19,6.20, and6.21and letting
λ(u,k)= |P(u,k)|
|N(u,k)| = |P(u)|
|N(u)| =λ(u) (6.31) as the numbers of positive and negative patterns used during the testing stage do not change for each fold and for each user, inequality6.30may be written as
MAE(u,k)≥FNR(u,k)× λ(u)
λ(u)+1 + FPR(u,k)× 1
λ(u)+1. (6.32) Given that
MAE(u)= 1 Kk∈[K]
MAE(u,k), (6.33)
inequality6.32may be written as:
MAE(u)≥FNR(u)× λ(u)
λ(u)+1 + FPR(u)× 1
λ(u)+1. (6.34) If we consider the average value for the MAE over all users, we may write that:
MAE= 1
|U|u∈UMAE(u). (6.35)
This results in:
MAE≥ 1
|U|u∈UFNR(u)× λ(u)
λ(u)+1 + FPR(u)× 1
λ(u)+1 (6.36) The worst case scenario concerning the classification performance of the second level is that the second-level (multi-class) classifier incorrectly assigns all true pos-itive patterns to C3 while they truly originated fromC1. In addition, all the false negative patterns originate fromC3and all the false positive patterns are assigned to C3. Thus, we may write the following inequality:
∀u∈U∀f ∈ [K]MAE(u,k)≤ 3×FN(u,k)+2×TP(u,k)+3×FP(u,k) P(u,k)+N(u,k)
(6.37) Given Eqs.6.18,6.19,6.20,6.21and6.31, Eq.6.37may be written as:
MAE(u,k)≤ 3×FNR(u,k)×λ(u)
λ(u)+1 + 2×TPR(u,k)×λ(u)
λ(u)+1 + 3×FPR(u,k) λ(u)+1
(6.38) Now, given Eq.6.33, inequality6.38results in:
MAE(u)≤ 3×FNR(u)×λ(u)
λ(u)+1 + 2×TPR(u)×λ(u)
λ(u)+1 + 3×FPR(u) λ(u)+1 (6.39) Thus, the average value for the MAE has an upper bound given by the following inequality: Inequalities6.36and6.40imply that the minimum value for the average MAE over all users is given as:
minu∈UMAE= |U1|
u∈U
FNR(u)×λ(u)
λ(u)+1 + FPRλ(u)+(u1). (6.41) Similarly, the maximum value for the average MAE over all users is given as:
maxu∈UMAE= |U1|
1. Breese, J.S., Heckerman, D., Kadie, C.: Empirical analysis of predictive algorithms for col-laborative filtering. In: Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence, pp. 43–52. Morgan Kaufmann (1998)
2. Pennock, D.M., Horvitz, E., Lawrence, S., Giles, C.L.: Collaborative filtering by personality diagnosis: a hybrid memory and model-based approach. In: Proceedings of the 16th Conference on Uncertainty in Artificial Intelligence, UAI’00, pp. 473–480. Morgan Kaufmann Publishers Inc., San Francisco (2000)
Chapter 7
Evaluation of Cascade Recommendation Methods
Abstract The experimental results provided in this chapter correspond to the test-ing stage of our system. The evaluation process compared three recommendation approaches: (a) the standard Collaborative Filtering methodologies, (b) the Cascade Content-based Recommendation methodology and (c) the Cascade Hybrid Recom-mendation methodology. To evaluate our system, we tested its performance as a RS for music files. In the following sections of this chapter, a detailed description is provided of the three types of experiments that were conducted in order to evaluate the efficiency of our cascade recommendation architecture.
7.1 Introduction
The evaluation process involved three recommendation approaches:
1. The first approach corresponds to the standard collaborative filtering methodolo-gies, namely the Pearson Correlation, the Vector Similarity and the Personality Diagnosis.
2. The second approach corresponds to the Cascade Content-based Recommenda-tion methodology which was realized on the basis of a two-level classificaRecommenda-tion scheme. Specifically, we tested one-class SVM for the first level, while the second classification level was realized as a multi-class SVM.
3. Finally, the third approach corresponds to the Cascade Hybrid Recommendation methodology which was implemented by a one-class SVM classification compo-nent at the first level and a CF counterpart at the second level. Specifically, the third recommendation approach involves three different recommenders which correspond to the different CF methodologies that were embedded within the second level.
Three types of experiments were conducted in order to evaluate the efficiency of our cascade recommendation architecture.
• The first type of experiments is described in Sect.7.2and demonstrates the con-tribution of the one-class classification component at the first level of our cascade recommendation system. Specifically, we provide MAE and RSC measurements
© Springer International Publishing Switzerland 2015
A.S. Lampropoulos and G.A. Tsihrintzis,Machine Learning Paradigms, Intelligent Systems Reference Library 92, DOI 10.1007/978-3-319-19135-5_7
111
concerning the mean overall performance of the standard collaborative filter-ing methodologies in the hybrid recommendation approach for the complete set of users. Additionally, we measure the relative performance of the Cascade Content-based Recommender against the performance of other recommendation approaches in order to identify the recommendation system that exhibits the best overall performance.
• The second type of experiments is described in Sect.7.3and demonstrates the contribution of the second (multi-class) classification level within the framework of the Cascade Content-based Recommendation methodology. The main purpose of this experimentation session is to reveal the benefit in recommendation quality obtained via the second (multi-class) classification level.