Regularized Fuzzy c-Means Clustering

Regularized Fuzzy c-Means Clustering Algorithm

The presented algorithm performs a topology preserving mapping from the high, n-dimensional space of the cluster centers onto a smaller,q≪n-dimensional map [193]. The cluster centers of thisq-dimensional map are connected to the adjacent cluster centers by a neighborhood relation, which dictates the topology of the map. For instance, Figure 2.18 shows a regular square grid, corresponding toc= 9

z

z

v

v

v

v

v

v

v

v

v

Figure 2.18: Example of cluster centers arranged in a two-dimensional space.

cluster centers. In this section,cis assumed to be known, based on prior knowledge, for instance. For methods to estimate or optimize c refer to [26]. This kind of arrangement of the clusters makes the result of the clustering interpretable only if the smoothness of the distribution of the cluster centers on thisq-dimensional map

is guaranteed (some illustrative examples will clarify this idea, see, e.g., Figure 2.19 and Figure 2.20). The neighborhood preserving arrangements of the cluster centers means the smoothness of the distribution of the cluster centers on this map. The aim of the visualization of the cluster centers is to provide easily interpretable maps of the variables, where every map is colored according to the numerical values of the cluster centers. In case of distance preserving arrangement it is expected that the resulted maps will be smooth, since clusters that are near to each other in the high-dimensional space of the original data have similar numerical values on each variable. Hence, the idea behind the presented algorithm is to cluster the data and simultaneously provide the smoothness of the resulted maps, of which the regularization indirectly ensures the neighborhood preserving properties of the arrangement.

The smoothness of the grid of the cluster centers can be measured by S=! """" This formula can be approximated by the summation of second-order differences, expressed in the following matrix multiplication form:

S≈tr(VG^TGV^T) =tr(VLV^T), (2.18) where G= [G^T₁,· · ·G^T_q]^T, where Gi denotes the second-order partial difference operator in thei= 1, . . . , qth direction of the map. On the boarders, the operator is not defined. In case of the model depicted in Figure 2.18, these matrices are the following:

To obtain ordered cluster centers, this smoothness measure can be added to the original cost function of the clustering algorithm (1.16)

J(X,U,V) = Taking into account constraints (1.12), (1.13) and (1.14), the minimum of (2.20) can be solved by a variety of methods. The most popular one is the alternating optimization algorithm [39] (see also Section 1.5.4). For fixed membership values, U, the minimum of the cost function with respect to the cluster centers,V, can be found by solving the following system of linear equations that can be followed from the Lagrange multiplier method:

vi

Based on the previous equation, the regularized FCM-AO algorithm is in Al-gorithm 2.3.1.

Relation to the Fuzzy Curve-Tracing Algorithm

The fuzzy curve-tracing algorithm [291] developed for the extraction of smooth curves from unordered noisy data can be considered as a special case of the previ-ously presented algorithm. The neighboring cluster centers are linked to produce a graph according to the average membership values. After the loops in the graph have been removed, the data are then re-clustered using the fuzzy c-means algo-rithm, with the constraint that the curve must be smooth. In this approach, the measure of smoothness is also the sum of second-order partial derivatives for the cluster centers:

As this cost-function shows, the fuzzy curve-tracing applies a clustering algorithm that can be considered as a special one-dimensional case of the presented regular-ized clustering algorithm (2.20), whereG1is the second-order difference operator.

Ifc= 5 and the clusters are arranged as can be seen, e.g., in Figure 2.18, thenG1

Algorithm 2.3.1 (Regularized Fuzzy c-Means).

• Initialization

Given a set of data Xspecify c, choose a weighting exponent m >1 and a termination tolerance ǫ > 0. Initialize the partition matrix such that (1.12),(1.13)and (1.14)holds.

• Repeat forl= 1,2, . . .

• Step 1: Calculate the cluster centers.

v^(l)_i = N k=1

µ^(l−1)_i,k m

x_k−ς c j=1,j=i

Li,jv_j N

k=1

µ^(l−1)_i,k m

+ςLi,j

,1≤i≤c. (2.22)

• Step 2 Compute the distances.

d²(xk,v_i) =

x_k−v_i^(l)T

A

x_k−v^(l)_i

1≤i≤c,1≤k≤N. (2.23)

• Step 3: Update the partition matrix.

µ^(l)_i,k= 1

c

j=1(d(x_k,v_i)/d(x_k,v_j))^2/(m−1), (2.24) 1≤i≤c,1≤k≤N

until||U^(l)−U^(l−1)||< ǫ.

Note that the elements ofLare identical to the regularization coefficients derived

“manually” in [291]. As the presented approach gives compact and generalized approach to the fuzzy curve-tracing algorithm, we can call our algorithm as Fuzzy Surface-Tracing (FST) method.

Relation to the Smoothly Distributed FCM Algorithm

The aim of this section is to illustrate that the smoothness of the grid can be defined on several ways. One possible implementation of smoothness (as used, for example, in spline theory) is achieved by demanding in an overall fashion that the value of a code vector must be close to the average value of its nearest neighbors on the grid. This interpretation has been used in [211]. Referring to Figure 2.18,

this means thatGi becomes a first-order gradient operator [211]:

This type of penalizing is not always advantageous, since the cluster centers get attracted to each other and thereby to the center of the data, where the magnitude of this contraction is determined by the regulation parameterς.

In the following the presented algorithm is used to trace a spiral in three-dimensional space and to show how the wine data can be visualized.

Example 2.7 (Tracing a Spiral in 3D based on regularized Fuzzy c-Means algo-rithm). The aim of the first example is to illustrate how increasing the smoothness of the of the grid of the cluster centers can generate topology preserving arrange-ment of the clusters. The illustrative problem that the regularized fuzzy clustering algorithm has to handle is to trace a part of a spiral in 3D. For this purpose 300 datapoints around a3D curvature have been clustered, where the datapoints were superposed by noise with 0 mean and variance 0.2. Both FCM and its regular-ized version are used to generate seven clusters that can be lined up to detect the curvature type structure of the data.

As can be seen from the comparison of Figure 2.19and Figure 2.20, with the help of the presented regularized clustering algorithm the detected cluster centers are arranged in a proper order.

In this example it was easy to qualify the result of the clustering, since the clusters and the data can be easily visualized in3D. This allows the easy tune of the only one parameter of the algorithm, ς, since with the increase of this param-eter it is straightforward to validate the ordering of the resulted clusters (as it is

x1

x2

x3

Figure 2.19: Detected clusters and the obtained ordering when the standard FCM algorithm is used.

x2

x1

x3

Figure 2.20: Detected clusters and the obtained ordering when the regularized FCM algorithm is used.

depicted in Figure2.20). This result shows that with the use of a properly weighted smoothingς= 2, it is possible to obtain proper arrangement of the cluster centers.

The aim of the next example is to illustrate how this kind of arrangement of the cluster centers can be obtained and used for the analysis of high-dimensional data.

Example 2.8 (Clustering and Visualization of the Wine data based on regular-ized Fuzzy c-Means clustering). For the illustration of the results of the smoothed clustering, the MATLAB implementation of the algorithm – which can be down-loaded from the website:www.fmt.vein.hu/softcomp– generates independent plots for each variables. On these plots the numerical values of the cluster centers are color coded, see Figure2.21.

Alcohol Maleic acid Ash Alc. ash Mg

Tot. Phenols Flavanoids Non−flav.Phen. Proan. Color int.

Hue OD280/OD315 Proline Label

3 3 3 2 2

3 0 2 2 2

3 2 2 2 2

0 1 1 1 1

1 1 1 1 1

Figure 2.21: Results of regularized clustering.

For the illustration how these plots can be interpreted the clustering and the visualization of the well-known wine data is considered.

From Figure 2.4 in Example 2.1 the different wines cannot be distinguished, and the relations between the variables and the origin of the wine are hard to get.

With the use of regularized fuzzy clustering three different regions corresponding to the tree different types of wine can be obtained, see Figure 2.21. The three regions can be seen on the maps of alcohol, color intensity, flavanoids, and ash.

This means, these variables have distinguishable value, hence a classifier system should be based on these variables.

For comparison, the maps obtained with the Kohonen’s SOM can be seen in Figure2.11 in Example 2.3. The FCM based algorithms found the same clusters

as Kohonen’s SOM and the same relations reveal. The SOM and the presented clustering algorithm can be effectively used for “correlation hunting”, that is, in-specting the possible correlations between vector components in the data. These procedures can also be useful for detecting the redundant features.