Color Processing
3.6 Feature selection methods
The selection of features is a very active area in recent years, especially in the context of data mining. Indeed, the data mining in very large databases is becoming a critical issue for applications such as image processing, finance, etc. It is important to summarize and intelligently retrieve the "knowledge" from raw data. The data mining is an area based on statistics, machine learning and the theory of databases.
The variable selection plays an important role in data mining especially in the preparation of data prior to processing. Indeed, the interests of the variable selection are as follows:
• When the number of variables is just too great so that learning algorithm cannot finish in a reasonable time. The selection reduces the dimension of feature space.
3.6. Feature selection methods 35
Figure 3.6: A framework for color space selection
Figure 3.7: Feature selection architecture
• In terms of artificial intelligence, creating a classifier returns to cre-ate a model for the data. However, a legitimate expectation for a model is to be as simple as possible (principle of Occam’s razor [Anselm Blumer Andrzej Ehrenfeucht 1987]).
Reducing the size of the feature space allows us to reduce the number of required parameters for the description of this model also avoiding the phenomenon of over-fitting and emphasizing the synthesize information.
• It improves the performance of the classification, its speed and power of gen-eralization.
• It increases the data understanding: a better view of what are the processes that give rise to them. This selection consists of:
– The elimination of independent variables of the class, – The elimination of redundant variables.
3.6.1 Global concept
A general structure for selecting features can be offered in the way of figure 3.7 ([Hall 1998]). Up to a certain criterion to be satisfied, subsets are generated in browsing the feature space. The subsets generation is a searching process in the subset space of cardinality2N with N the number of features. All classical searching algorithms can be applied to that problem. For instance [Koller 1996] proposes the methods forward addition and backward elimination (deletion), [Dangauthier 2005]
and [Yang 1998] have made a good use of evolutionary algorithms.
3.6.2 Searching algorithm and evaluation
Existing feature selection methods for machine learning typically fall into two broad categories : those which evaluate the worth of features using the learning algorithm that is to ultimately be applied to the data, and those which evaluate the worth of
3.6. Feature selection methods 37
features by using heuristics based on general characteristics of the data. The former are referred to as wrappers and the latter filters.
1. Wrappers use classification algorithm to evaluate the pertinence of a given subset of variables.
2. Filters are completely independent from the classification stage. They are based on statistical concepts: entropy, coherence . . . A good feature subset is one that contains feature highly correlated with predictive of the class and yet uncorrelated with the others [Hall 1998].
The wrappers: Although conceptually more simple than filters, wrappers were introduced more recently by John, and Kohavi Pfleger in 1994. Their principle is to generate subsets candidates and to evaluate them thanks to a classification algorithm. The score or merit will be a combination of a trade-off between the number of variables eliminated, and the classification rate on a test file. Thus, the
“assessment” stage of the selection cycle is made by a call to the classification algo-rithm. In fact, the classification algorithm is called several times for each evaluation because a cross-validation is frequently used. By its very intuitive principle, this method generates subsets well suited to the classification algorithm. Recognition rates are high since the selection takes into account the intrinsic bias of data. An-other advantage is its conceptual simplicity: there is no need to understand how the induction is affected by the selection of variables, it is sufficient to generate and test. However, there are three reasons that the wrappers are not a perfect solution.
First, they do not really have theoretical justification for the selection and they do not allow us to understand the conditional dependencies that may exist between the variables. On the other hand, the selection process is specific to a particular classification algorithm and found subsets are not necessarily valid if you change the method of induction. Finally, and this is the main defect of the method, the calculations quickly become quite long when the number of variables grows up.
The filters: Filters don’t have the defects of wrappers. They are much faster, they are based on more theoretical considerations, it allows a better understanding to the dependency relationships between variables. But, as they do not take into account the biases of the classification algorithm, the subsets of variables generated give a lower recognition rate. To give a score to a subset, the first solution is to give a score to each variable independently of the others and to do the sum of those scores (OneR Selection). The alternative is to evaluate a subset as a whole [Dangauthier 2005]. However, there is an intermediary between ranking and feature subset ranking based on an idea of Ghiselli and used with good results in the context of the CFS (correlation based feature selection) by M.A. Hall [Hall 1998]. The score of a subset is constructed based on correlations variable-class and correlations variable-variable (Eq.3.5):
rzc = k.rzi pk+k(k−1)rii
(3.5)
Whererzc is the correlation between the summed components and the outside vari-able (a given color cluster – a class), k is the number of components, rzi is the average of the correlations between the components and the outside variable, and rii is the average inter-correlation between components. This equation expresses that the merit of a given subset increases if the variables are highly correlated with the class and it decreases if features are highly correlated between each others. The idea is to state that a "good" subset is composed of variables highly correlated with the class (to discard independent variables) and loosely correlated between them/features (to avoid redundant components). It is an approximation since it only takes into account the interactions of order 1. The correlation or dependency between two variables can be defined in several ways. Using the statistical corre-lation coefficient is too restrictive because it only captures the linear dependence.
However, one can use a test of independence as the statistical test of χ2. It is also possible to combine wrapper and filter as presented in [Yang 1998].
Stopping criterion Finally, the stopping criterion may take various forms: a computation time, a number of generations (in the case of a genetic algorithm for instance), a number of selected variables or a heuristic evaluation of the subset
"value".
3.6.3 Summary on feature selection methods in use
After this short review on feature selection methods, we propose to categorize and describe the approaches which are used in our color space selection framework. These different methods are mentioned because they cover the main types of attribute selection algorithm.
Hybrid color space built by genetic algorithm. GACS Basics:
• Attribute Evaluator: 1-Nearest Neighbor classifier (1-NN)
• Search method: Genetic search
Genetic algorithm (GA) is a search technique used in computing to find exact or approximate solutions to optimization and search problems. Genetic algorithms are categorized as global search heuristics and are a particular class of evolutionary algorithms (EA) that use techniques inspired by evolutionary biology such as inher-itance, mutation, selection, and crossover. Full details about Genetic algorithms in search, optimization and machine learning are presented in [Goldberg 1989].
In Hybrid Color Space (HCS) context, each individual has to encode a vector, where each component is an axis of the HCS. We consider a set C of features, C ={Ci}Ni=1 ={R, G, B, I1, I2, I3, L∗, u∗, v∗, ...} with Card(C)=25. Practically, it is almost impossible to test all possible combinations; the number of feasible so-lutions evolves according a factorial function of the total number of the candidate primaries (combinatorial explosion). Hence, GAs are well suited to get rid off absurd
3.6. Feature selection methods 39
Figure 3.8: Cross over operator
combinations. Roughly, the first step is to initialize the population, each individual is made up picking randomly three elements of C. Concerning cross over operator, two individuals h1 and h2 share their genetic material, swapping one of their com-ponent; figure3.8. Finally, to perform mutation on an individual, one component is selected and replaced at random by an element of C. Finally, the evaluation phase computes a1-NN classifier based on a Euclidian metric.
Correlation-based Feature Subset Selection (CFS) Basics:
• Attribute Evaluator: Statistical
• Search method: Greedy stepwise
Correlation-based Feature Subset Selection (CFS) proposed by M.A. Hall [Hall 1998]
evaluates the worth of a subset of attributes by considering the individual predictive ability of each feature along with the degree of redundancy between them. Subsets of features that are highly correlated with the class while having low intercorrelation are preferred. The greedy stepwise search method performs a greedy forward search through the space of attribute subsets. It may start with no attributes and stops when the addition of any remaining attributes results in a decrease in evaluation.
The cardinality of the final subset can be set to three in our case.
De-correlated Hybrid Color Space (DHCS) Basics:
• Attribute Evaluator: Statistical
• Search method: Ranker
The basic idea of a De-correlated Hybrid Color Space
[J. D. Rugna P. Colantoni 2004] is to combine different color components from different color spaces. Considering that there is a high redundancy between colors components it is, in a general way, quite difficult to define criteria of analysis to compute automatically the most relevant color components corresponding to a selected set of color components. That is the reason why, in order to build a hybrid color space, based on K’ color components, from K selected color components,
Name Type Evaluation Searching algorithm CFS Filter Statistical Greedy stepwise
DHCS Filter Statistical Ranker
GACS Wrapper Classification Genetic Algorithm OneRS Wrapper Classification Ranker
Table 3.2: Selection feature methods in use
such as K’ << K, the proposed method: (1) computes the covariance matrix (of size KxK) of K color components selected, (2) computes the eigenvectors and the eigen values of this matrix, (3) reduces to K’ the number of color components in computing the K’ most significant eigen values of the covariance matrix from a principal component analysis (PCA). It can also produce a ranked list of attributes by traversing the space from one side to the other and recording the order in which attributes are selected.
One Rule Selection method (OneRS) Basics:
• Attribute Evaluator: One Rule Evaluation
• Search method: Ranker
The One Rule Selection method classes for building and using a One-R classifier; it evaluates the worth of an attribute by using the OneR classifier, in other words, uses the minimum-error attribute for prediction, discretizing numeric attributes. OneR, short for "One Rule", is a simple classification algorithm that generates a one-level decision tree. OneR is able to infer typically simple, yet accurate, classification rules from a set of instances. The OneR algorithm creates one rule for each attribute in the training data, then selects the rule with the smallest error rate as its ’one rule’.
To create a rule for an attribute, the most frequent class for each attribute value must be determined. The most frequent class is simply the class that appears most often for that attribute value. A rule is simply a set of attribute values bound to their majority class; one such binding for each attribute value of the attribute the rule is based on. The error rate of a rule is the number of training data instances in which the class of an attribute value does not agree with the binding for that attribute value in the rule. OneR selects the rule with the lowest error rate. Finally, it ranks attributes according to error rate (on the training set). It treats all numerically-valued attributes as continuous and uses a straightforward method to divide the range of values into several disjoint intervals. A full report on this selection method can be found in [Holte 1993]. A summary of the different methods is reported in table 3.2.
3.7. Image Segmentation for Hybrid Color Spaces: Vector Gradient 41