E. Blanco, M. Mazo, L.M. Bergasa, S. Palazuelos and A.B. Awawdeh University of Alcalá/ Department of Electronics, Alcalá de Henares, Spain
edward@depeca.uah.es, mazo@depeca.uah.es, bergasa@depeca.uah.es, sira@depeca.uah.es, abdel@depeca.uah.es Abstract – This paper presents an alternative that allows to
improve the color image segmentation in the HS sub-space (HSI space). The authors propose to apply a zero order transformation in the RGB space which consists in adding a vector in the RGB space to control the separation between classes in the HS sub-space. This vector is considered optimum. To define it, the chromatic C1C2sub-space (YC1C2 space) is used. The proposal presented in this work has been designed to be applied in real-time on each consecutive frame of a sequence of color images.
The effectiveness of this work has been tested and verified using applications where a reduced contrast between the background color and the color of the object to segment exists, and when the size of the object to segment is very small in comparison with the size of the captured scene. Furthermore, the process of segmentation is improved and, at the same time, the effects of the variations of the light intensity of the scene are considerably reduced.
I. INTRODUCTION
From the recent works on color segmentation we would like to emphasize those related to the segmentation of the skin natural color. In this field, S.L. Phung et al. [1] segmented the skin using a Bayesian classifier, obtaining satisfactory results even under adverse illumination conditions for different color spaces such as: HSV, RGB, YCbCr and CIE-Lab. R.-L. Hsu et al. [2] suggest the detection of face skin, considering a nonlinear subspace of YCbCr space that allows to compensate part of the luminosity variations. The trend to compensate part of these luminosity changes is the use of dynamic color models as done by L. Sigal et al. [3]. L. Sigal presents an overview of the works and researches done in the color skin segmentation field.
The most frequently used color spaces in this type of applications are HSV [3], [4] and normalized rg [5]-[7]. The perceptive color spaces as HSV and HLS are widely used in image processing, as well as HSI, where, in some works, only the Hue (H) and Intensity (I) components are used in the clustering process [8]. In other cases, as in [9], a threshold value for the Saturation (S) of each pixel based on its intensity is defined. This threshold is used before the clustering process to determine if S should be replaced by H or I.
In the sign language recognition field, N. Habili et al. [10]
performed a pixel-by-pixel classification of the skin color with discriminant features of the CbCr plane, using the Mahalanobis distance, but he needs a fusion of motion cues to obtain good results. A similar segmentation is achieved in the work done by D. Chai et al. [11], where post-segmentation process steps have been applied, such as morphological
operations, in order to surpass the limitations of the segmentation. In [11], the YCbCr space has been used too.
This color space is one of the most widely used in the segmentation process.
In the sign language recognition field, it is very important to detect the geometric form of the object to segment (face and hands edges). This is the reason why it is necessary to use strategies to enhance the contrast between the object and the background of the scene. This work aims to enhance the contrast by means of a class separation pre-process in the HSI space [12], by properly coloring the image with a color vector in the RGB space, taking into account the YC1C2 space [8], [13], [14].
This paper has been organized as follows: section II describes a general vision of the proposed algorithm to increment the separation between classes. Section III presents the criteria considered when separating the classes. Section IV details how to improve the separation between classes in HS plane starting from their location in the C1C2 plane. Section V presents the algorithm that performs the optimal class separation. Section VI describes how to obtain the color vector to add, and the effects it produces in the images. Section VII contains the experimental results, and section VIII the conclusions and future works.
II. ALGORITHM OVERVIEW
As mentioned before, the objective of this work is to improve the segmentation process by adding an optimal color vector to each one of the captured images in the RGB space.
This vector is different from one image to another, looking for the maximum separation between classes in the HS plane (sub-space where the segmentation must be performed).
As it is known, an important property of the HSI space (perceptive space) is that it produces a maximum disconnection between the chrominance and luminance components. As a result, the luminance can be almost fully isolated, making the segmentation process more invariant to the changes in shades and illumination, as in [4]. For this reason, in our proposal, the analysis in the HSI space only considers the H and S chromatic components (HS plane).
If the original image is denoted by I, the optimal color vector to add by ir, and the colored image, resulting of the addition, by Ii, is fulfilled:
ir
i= +
I I . (1)
The determination of the optimal color vector ir, for each captured RGB image, is done following the steps shown in
125
BLANCO ET AL . Fig. 1. These steps are:
1) From the captured RGB image (I), several significant samples (seeds) are obtained from both the object (class O) and the background (class B). We will refer to the object class in the RGB space as ORGB={rOk} k=1,2,..,N, while BRGB={rBq} q=1,2,..,M, will refer to the background class, where N and M are the number of pixel seeds taken for classes O and B, respectively.
2) For every ORGB and BRGBsamples (in the RGB space), a transformation to the YC1C2 space is done. After this transformation, the resulting classes will be referred to as OC1C2={cOk} and BC1C2 ={cBq}.
3) Making use of the properties and relationship between the HSI and YC1C2 color spaces, the optimal location of the classes in the C1C2 space is obtained, by finding the optimal location of their respective mean vectors, ciOopt and ciBopt.
4) From the ciOopt and ciBopt mean vectors, its corresponding ones in the RGB space, riO and riB, can be calculated. With these RGB vectors and the mean vectors of the original classes (ORGB, BRGB) represented by rO and rB, the optimal color vector can be calculated, as shown in (2) in the case of the O class:
r iO O
i =r −r . (2)
5) Once the optimal color vector has been obtained, the new colored image Ii can be calculated using (1).
Finally, the colored image Ii is transformed from the RGB space to the HS plane, where the segmentation is done. The color vector ir has its effects in the HSI space due to its nonlinearity.
III. CRITERIA FOR THE SEPARATION BETWEEN CLASSES
The Fisher Ratio (FR) is frequently used to measure the class separability in classification systems [15], [16]. This ratio simultaneously quantifies the inter-class and intra-class scatter. For a two-class system, it is interesting to achieve a large metric distance between the class means and a minimum dispersion within each class (leading to a high FR). In this work, the FR is used as a measurement index of the effectiveness of the separation between classes for pixel classification in the HS plane. In the case of a multi-class system, the generalized FR [17] is expressed by:
FR=trace(Mw-1Mb), where Mb refers to the inter-class (between class) covariance matrix and Mw refers to the
internal (within class) dispersion matrix of the class. Due the circular form of the trajectory of H component, this last equation cannot be directly applied. The reason is that for a two-class system (our case), the Mb does not represent a real distance between the H means of both classes. In addition, the Mw does not represent a real H variance in the discontinuity point when H moves from 2π to 0 in its trajectory. In order to solve this problem, a particular FR has been calculated separately [17] for each component in the HS plane, given by:
H S
FR=FR +FR (3)
where FRH and FRS represent the Fisher Ratio of the H and S components, respectively, given by (4):
2 2
H h HO HB
FR =θ σ +σ , FRS=||SO−SB|| σSO2 +σSB2 (4) where θhis the separation angle between the hue means of both classes, ||SO-SB|| is the distance between the saturation means of both classes, σSOand σSB are the standard deviations of the saturation for both classes, and σHO and σHB are the standard deviations of the hue.
IV. SEPARATION OF THE CLASSES IN THE HSPLANE
STARTING FROM THEIR LOCATION IN THE C1C2PLANE
This section details the most important relationships between the statistical mean and variance of the classes in the C1C2 and HS planes. Also, the effect of an increment in two vectors in RGB space on the projections of these vectors in the C1C2 and HS planes is analyzed.
A. Relationships between the RGB Space, and the HS and C1C2
Planes
Given a vector r = [R G B]T located in the RGB space, the HS components of a vector h [12] given by:
(1 3min( , , ) ( )) obtained from r by:
1
where Q is the transformation matrix between spaces.
From (7), the components C1C2 of the vector c are:
Therefore, using (6) and (10) it can be demonstrated that a vector in the RGB space can be projected in the HS and C1C2
planes with the same phase shift but a different module, that is: H=H’ and S≠C. It can also be demonstrated that the relationship between S and C is:
Fig. 1. General block-diagram of the proposed algorithm to define the optimal color vector.
126
IMPROVEMENT OF SEGMENTATION IN HS SUB-SPACE where I is the intensity (HSI space), which coincides with Y component, f(H) is a weighting function that depends on the H component and f(H) ∈ [0.5, 1]. This f(H) function generates a three lobe curve in the HS plane delimitated by the discontinuities corresponding to the three color sectors of the plane: 0-2π/3, 2π/3-4π/3 y 4π/3-2π.
After all the above discussion, given two vectors in the RGB space, rO and rB,the resulting projection vectors in the C1C2 (cO and cB), and HS planes (hO and hB), it is fulfilled: intensity means of both classes, object and background, respectively, and corresponding to the hO and hB vectors.
On the other hand, it is important to note that when adding a vector ir to both rO and rB in the RGB space, the distance vector dc=cO-cB in the C1C2 plane remains constant, so its magnitude (||dc||) and orientation (φ) are invariant. Therefore, adding ir in the RGB space results in a translation of the classes in the C1C2 plane. This effect can be achieved with a translation vector ic (corresponding to ir) directly added in the C1C2plane. more difficult to model it. This is so, because dh depends (14) on the value of IO and IB and on the f(H) function. In any case, (12) always holds.
Summarizing: to obtain the value of the color vector to be added in the RBG space, the authors suggest the use of the relationship between the h vector components in the HS plane, and their corresponding c vector components in the C1C2
plane, given by (10) and (11), and the relationship between pairs of vectors in these planes, given by (12, 13 and 14).
Therefore, the proposed method is based on the analysis of the cO and cB vectors in the C1C2 plane and the properties of the difference vector dc (invariants).
B. Separation between the Means of Hue (Angular Separation) The possibility of utilizing dh to obtain the separation between the hue means is rejected due to the discontinuities presented by dh because (14) is a function of (11). It is proposed to use C1C2 plane, where the distance function between the vectors ciO and ciB, (||dc||) (13) does not present discontinuities. Therefore, the proposed algorithm has been
parameterized as a function of the separation angle (θi) between the vectors already added with ir, ciO and ciB (“i” indicates that the color vector has been added).
In our case, the optimal angle θi is obtained from an observation function that measures the effectiveness of the separation between the classes in different locations in the HS plane. This function will be described in paragraph e of section V.
When the angle of separation θi is maximum, θi coincides with the angle whose bisector is a straight line p, that passes through the origin of coordinates and is perpendicular to the straight line, l, whose director vector is dc (Fig. 2). Therefore, the vector to add (ir) that causes the maximum difference of hue, makes the modules of both vectors ciO and ciB be equal (||ciO||=||ciB||) and the distance between the intersection point of the lines p and l, and the extreme of each vector be ||dc||/2 (forced location). In Fig. 2, an example of the forced location of the vectors cO and cB after the addition of the color vector (ciO and ciB) is shown.
C. Separation between the Means of Saturation
Given two vectors, for example hiO and hiB, in the HS plane, we analyze how the value of the difference of saturation, SO-SB=||hiO||-||hiB||, varies when θi between both vectors changes. In our case as ||ciO||=||ciB||=Ci, according to (11), the intensities (IO, IB) corresponding to both vectors hiO
and hiB, and the value of the saturation weighting function f(H) of each one, are the parameters that have a significant effect in the value of SO-SB. It is important to note that the saturation varies inversely with the intensity. According to this, it can be said that SO-SB is determined by: a) the intensities of the vectors hiO and hiB (IO, IB), and b) the module and phase of dc (the invariants) since these ones determine the location of the vectors hiO and hiB along the curve f (H) in the HS plane.
D. Analysis of the Dispersion of the Classes
In this paragraph we carry out an analysis of how the dispersions of saturation and hue of the classes in the HS plane are affected when they are translated in the C1C2 plane, as a result of the addition of the color vector (ir). This analysis will be necessary to obtain the class separation measure function.
1. Dispersion of the Hue (Angular Dispersion) Knowing that the variation of the angular dispersion in the
Fig. 2. Forced location of ciO and ciB vectors in the C1C2plane, once the color vector has been added.
127
BLANCO ET AL . C1C2 plane corresponds with the variation of the dispersion of
hue in the HS plane, and being C1C2 plane a Cartesian plane, the problem poses in the polar coordinates, taking these two considerations into account:
a) As it has already been indicated, in the C1C2 plane, the addition of ir produces translation of the classes and, therefore, variations of the modules of their mean vectors
||ciO||=||ciB||=Ci. This causes that the angular dispersion of both classes is modified. The angular dispersion increases as the module of its respective mean vector decreases, due to the increment of the separation angle θi, according to:
Ci=||dc||/2sin(θi/2).
b) The geometric forms of the distributions of the classes are not predetermined, but they can vary, since the samples are randomly taken from the object and the background, implying that the dispersion varies. The reason is that for different translations of a class in the C1C2 plane, different orientations between the axis of maximum and minimum dispersion (represented by their uncertainty ellipse in a C1C2 plane) with respect to the orientation of their mean vectors (ciO or ciB) are generated.
As an example, Fig. 3 shows both classes, before a translation (OC1C2 and BC1C2) and after it (OiC1C2 and BiC1C2,). It can be observed that the deviation of hue, σiHO, of OiC1C2 is greater than the deviation of OC1C2 (σHO), by the effect analyzed in the previous paragraphs (a and b). Nevertheless, the deviation of BiC1C2is lower than the deviation of BC1C2, by the effect analyzed in the a paragraph, since the module of ciB
is greater than the module of cB. 2. Dispersion of the Saturation
In fact, if all the vectors of the class have the same intensity, the dispersion of the saturation component is not directly affected by the effects of the translation of the classes in the C1C2 plane. The reason is that the saturation is a linear function of the components C1 and C2, as it can be seen in (15). It is possible to demonstrate that (15) is the saturation (11) particularized for lobe 1 of f(H).
3 3
1 2
S=C I C+ I (15)
This characteristic of linearity makes the deviation of the saturation (σS) constant, since the distance between vectors in C1C2 plane remains constant independently of the addition of ir. Nevertheless, in the HS plane σSwill be different for each lobe of f(H) but will stay constant within each lobe. Evidently, if the vectors of the class have different intensity, the
dispersion of the saturation will not be constant for each location, not even within the lobes (there is a greater variation of σS when the dispersion of the intensity component is greater).
V. ALGORITHM FOR THE OPTIMAL LOCATION OF THE
MEAN VECTORS OF BOTH CLASSES IN C1C2PLANE
This section presents the strategy used for the obtaining, in C1C2 plane, of the mean vectors ciOoptandciOopt that maximize the separation of the classes in the HS plane. The process consists of different phases, including an iterative algorithm to obtain a set of locations for the mean vectors of the classes (ciO and ciB) in C1C2 plane. The location of each vector will be parameterized by the angle formed between both vectors, θi. Therefore, we try to obtain a set of θin (θi1, θi2,…), each of them will have associated an index of measurement of separation between classes, that we will identify by βHSn (βHS1, βHS2,…). From the function βHSn=f(θin), the value of θin that produces the maximum separation between classes is obtained, θin optimal: θopt. The process begins obtaining in C1C2 plane the mean vectors of each class, i.e., cO=E{cOk} and cB=E{cBq}. The invariants of vector dc are obtained from the vectors cO and cB, i.e., ||dc||=(dC12+dC22)1/2 and φ=tan-1(dC2/dC1), where, dC1=C1O-C1B, dC2=C2O-C2B, and (C1O, C2O) and (C1B, C2B) are the components of the vectors cO and cB, respectively.
The iterative process consists of the following 5 steps:
a) Forced location of the mean vectors in the C1C2 plane The original vectors cO and cB are relocated (forced) in the C1C2 plane using the invariants (||dc||, φ), obtaining ciO and ciB
with the following restriction:
||ciO|| ||ciB|| dc 2sin( / 2)
i i
C = = = θ . (16)
The Cartesian components of these vectors (Fig. 2), particularized for the vector ciO, are given by:
cos( )
1iO i iO
C =C H , C2iO=Cisin(HiO) (17)
where HiO is the angle of the vector that can be expressed by:
2 2
iO i
H =π +φ θ− . (18)
The iterative algorithm is initialized with an initial θi equal to θ, where θis the angle formed by the vectors cO and cB. In each iteration (j) of the algorithm the value of θi, is increased:
θi(j)=θi(j-1)+Δθ.
b) Verification of the ciO and ciB vectors locations validity For each increase of θi, the validity of the locations of the vectors ciO and ciB is verified, checking if the values of the components of the corresponding vectors (riO, riB) in RGB space are lower than 1. If the locations are valid, the value of θi will be included in the set θin.
c) Calculation of the translation vector and translations of both classes in C1C2 plane
The translation vector ic is obtained for each value of θin. This ic is responsible of the translations of the classes from its original position to the forced location defined by θin. For the O class, this vector is given by: ic=ciO-cO. The translation of both classes in the C1C2 plane is made with ic. For the O class:
OiC1C2={cOk+ic}; k=1,2,..,N.
d) Classes transformation from the C1C2 plane to the HS
Fig. 3. Translation of the dispersions in the C1C2 plane and their different alignments with the mean vectors of each class.
128
IMPROVEMENT OF SEGMENTATION IN HS SUB-SPACE The classes in the HS plane (OiHS and BiHS) are obtained from
the translated classes OiC1C2 and BiC1C2, using (10), (15) and knowing that I=Y.
e) The observation function: calculation of the measurement index of the separation between classes (βHSn) in the HS plane As an observation function of the separation between the classes, a normalized index of measurement has been defined (βHS) from the FR described in (3). To obtain the βHSn
e) The observation function: calculation of the measurement index of the separation between classes (βHSn) in the HS plane As an observation function of the separation between the classes, a normalized index of measurement has been defined (βHS) from the FR described in (3). To obtain the βHSn