Point clouds results

Dans le document The DART-Europe E-theses Portal (Page 102-107)

5.2 Empirical analysis of the impact of Quality and Diversity to optimize a

5.2.2 Point clouds results

In this section, we analyze the influence of the diversity and the quality of the col-laborators on our proposed framework. In our case, we choose to use a local oriented entropy measure (Equation (5.10)) derived from the global entropy in Equation (4.25) as a diversity index, and both the Silhouette index and the Davies-Bouldin index as quality indexes. The experimental protocol is the following: we randomly modified already exist-ing solution vectors that we had found for all respective subsets and we analyzed the evolution of the Davies-Bouldin index and Silhouette index for 1 iteration of our col-laborative process using two or three of these random solution vectors collaborating together. The data sets are randomly split to have 2 or 3 algorithms working on dif-ferent attributes. A similar protocol is used with the VHR Strasbourg data set, but in addition to that, the number of cluster to be found being unknown the collaborators are looking for a different number of clusters. The algorithms used is the collaborative EM algorithm for all data sets. All experiments were realized with λ= 1− J1.

The results are shown in the form of four point clouds for all data sets:

• The raw Silhouette index difference between the two collaborators depending on the initial diversity.

• The Silhouette index raw improvement depending on the initial entropy between the collaborators.

• The Silhouette index raw improvement depending on the initial Silhouette index raw difference between the two collaborators.

• The Bouldin index raw improvement depending on the initial Davies-Bouldin index raw difference between the two collaborators.

Note that the first point cloud is not an experimental result and is just here to show that our simulation covered most possible cases of differences in quality and diversity prior to the collaborative process.

We used point clouds containing 1500 simulated collaborations (3000 points since the collaboration is evaluated both ways) for the Iris, WDBC, Wine and Image data sets, and 250 simulated collaborations (500 points) for the VHR Strasbourg data set because of computation time issues. The point clouds for these experiments are shown in Figures 5.3, 5.4, 5.5, 5.6 and 5.7.

We first want to analyze Figures 5.3(a), 5.4(a), 5.5(a), 5.6(a) and 5.7(a). These figures show the initial characteristics of the collaborators before collaboration by com-paring the initial quality difference using the silhouette Index with the initial diversity.

As one can see, in the case of the easy data sets -small in size and with few attributes-we attributes-were able to generate random example that are representative of the full range of possible couples, see Figures 5.3(a), 5.4(a) and 5.5(a). However, for more complex data sets we were unable to cover the whole range of possible couples, see Figures 5.6(a) and 5.7(a). Specifically, the random generator could not find clustering results that have a similar quality index but a high diversity. Nevertheless, we believe that our results are still significant even for complex data sets.

(a) ∆Silhouette=f(Diversity) (b) SilhouetteGain=f(Diversity)

(c) SilhouetteGain=f(∆Silhouette) (d) DBGain=f(∆DB)

Figure 5.3: Iris Data Set Point Clouds

(a) ∆Silhouette=f(Diversity) (b) SilhouetteGain=f(Diversity)

(c) SilhouetteGain=f(∆Silhouette) (d) DBGain=f(∆DB)

Figure 5.4: WDBC Data Set Point Clouds

We now move to the analysis of the influence of Diversity on the collaboration results. In figures 5.3(b), 5.4(b), 5.5(b), 5.6(b) and 5.7(b), we show how the value of the Silhouette Index evolved during the collaboration depending on the initial diversity between the two clustering solutions before collaboration.

The influence of diversity has already been the subject of an earlier study in our team, see [56], which after experiments similar to ours with another collaborative frame-work and supervised indexes concluded that a diversity around 50% has the best po-tential to give good collaboration results. However, as can be seen in our figures, we do not find the same results. While both studies agree that a low diversity induces a

5.2. EMPIRICAL MODEL BASED ON QUALITY AND DIVERSITY 103

(a)∆Silhouette=f(Diversity) (b) SilhouetteGain=f(Diversity)

(c) SilhouetteGain=f(∆Silhouette) (d)DBGain=f(∆DB)

Figure 5.5: Wine Data Set Point Clouds

small potential for a good collaboration, our experiments do not show any performance spike around 50%. In our case, a higher diversity always induces a stronger potential for better results. By potential we mean that the best results were always achieved when the diversity was very high, but that a high diversity does not always imply good collaboration results. These divergences between our empirical results, the results of previous empirical studies, and the theoretical results are discussed in Section 5.3.

(a)∆Silhouette=f(Diversity) (b) SilhouetteGain=f(Diversity)

(c) SilhouetteGain=f(∆Silhouette) (d)DBGain=f(∆DB)

Figure 5.6: Image Segmentation Data Set Point Clouds

Finally, we now move to Figures 5.3(c), 5.3(d), 5.4(c), 5.4(d), 5.5(c), 5.5(d), 5.6(c), 5.6(c), 5.7(c) and 5.7(d). In these figures, we show how the initial quality difference

(a) ∆Silhouette=f(Diversity) (b) SilhouetteGain=f(Diversity)

(c) SilhouetteGain=f(∆Silhouette) (d) DBGain=f(∆DB)

Figure 5.7: VHR Strasbourg Data Set Point Clouds

between two results before collaboration can influence the outcome of the collaboration.

We conducted this study using two indexes: the Silhouette Index and the Davies-Bouldin Index.

In term of global performances of our original algorithm, if we analyze our results using the diagram that we introduced in Figure 5.2, we can see that our proposed collaboration frameworks gives very good results: for all data sets, most of the collab-oration results fall into the good collaboration area: when receiving information from a better collaborator, the results after collaboration are almost always improves (fair collaboration) and in many cases they even beat the initially best collaborator (good collaboration). And when receiving information from a lower quality collaborator, the results are still improved.

We can see that for all data sets, most cases of negative and bad collaborations occur either when collaborating with solutions that have a similar or lower quality.

When we cross these results with these on the influence of diversity, we can highlight that these negative collaboration areas are found when both the diversity and quality of the collaborators are low. We can clearly see that collaborators that have a very low comparative quality but a high diversity give better collaborative results than those that also have a lower quality but a low diversity. Our explanation for this result is that given our collaborative model, in the case of a very weak collaborator having a high diversity (i.e. an almost random solution vector), the collaborative term looses all its influence and only the local algorithm term matters. This therefore results an improvement despite the negative collaborator. However, in the case of a weak collaborator but a low diversity (weak local result), the collaborative term still has a lot of influence in the process and thus may lead to a deterioration of the results.

We then applied the same protocol to have 3 random collaborators instead of only 2 working together, with the goal of determining whether or not the previous results would remain true with a higher number of algorithms. What we found out is that a higher number of collaborators increases the chances of conflicts and thus decreases the overall chances of achieving the best collaboration results, see Figures 5.8(b) and

5.2. EMPIRICAL MODEL BASED ON QUALITY AND DIVERSITY 105

(a) SilhouetteGain=f(Diversity) (b) SilhouetteGain=f(∆Silhouette)

Figure 5.8: Iris Data Set Point Clouds: 3 collaborators

5.9(b). Our interpretation is that we have two sub-cases when the collaborators are better on average than the local algorithm:

1. All collaborators are better than the local algorithm, in which case the results are the same than for the first part of the experiments with only 1 external collaborator. The result after collaboration in this case is better than the average results proposed by the collaborators, and it falls in the “good collaboration” area.

2. The collaborators are better than the local algorithm only on average: some of them are still weaker than the local algorithm and negatively influence the results, thus leading to weaker performances after collaboration. This issue could therefore be solved by giving a lower weight to weaker collaborators given a specific quality criterion.

However, despite slightly weaker performances, most properties that we found with 2 collaborators remained true with 3: a higher diversity still leads to potentially better results (Figures 5.8(a) and 5.9(a)), collaborating with algorithms that are weaker on average does not necessarily leads a negative collaboration as long as the diversity is high enough, and finally the results of our proposed framework remain satisfactory with most collaboration falling into the “good collaboration” or “fair collaboration” area. We found the same tendencies with all data sets.

(a) SilhouetteGain=f(Diversity) (b) SilhouetteGain=f(∆Silhouette)

Figure 5.9: EColi Data Set Point Clouds: 3 collaborators

The conclusions that we can draw from this experiment is that while our proposed framework is tolerant to weak collaborators and mostly negates their influence, the ideal conditions for good collaborations would require algorithms that already give high quality results, as well as a diversity as high as possible between these results.

Therefore most risks can probably be averted by detecting algorithms with solutions

that have low quality and diversity indexes. And we will show on what basis this can be done in the next section.

It would also be possible to reduce the computational complexity of the framework by removing them early in the collaborative process.

One possible weakness of this experiment is that it is not easy to determine whether the improvements on the results come mostly from the local term, the collaborative one, or both. A possible answer to this question could lie in Figures 5.6(b) and 5.6(c), as well as Figures 5.7(b) and 5.7(c), where we can see the the quality gain on the Silhouette index based on the initial diversity is a folded version of the quality gain depending on the initial quality difference. Based on these point clouds, and if we assume for a moment that all improvements with a weaker collaborator come only from the local term, the real influence of the collaborative term can be approximated on the diversity based diagrams and is still significant.

Dans le document The DART-Europe E-theses Portal (Page 102-107)