The Vapnik-Chervonenkis Dimension

The concept of the Vapnik-Chervonenkis dimension of a collection of sets was intro-duced in [6] and independently in [7]. Its main interest for data mining is related to one of the basic models of machine learning, the probably approximately correct learning paradigm as was shown in [8]. The subject is of great interest to probability theorists interested in empirical processes [9,10].

Definition 3.52 LetCbe a collection of sets. If the trace ofCon K ,CKequalsP(K), then we say that K is shattered byC.

The Vapnik-Chervonenkis dimension of the collectionC(called the VC-dimension for brevity) is the largest cardinality of a set K that is shattered byCand is denoted byVCD_(C).

IfVCD_(C)=d, then there exists a set K of size d such that for each subset L of K there exists a set C ∪Csuch that L =K ∩C.

Note that a collectionCshatters a set K if and only ifCKshatters K . This allows us to assume without loss of generality that both the sets of the collectionCand a set K shattered byCare subsets of a set U .

LetCbe a collection of sets withVCD_(C)=d and let K be a set shattered byC with|K| =d. Since there exist 2^d subsets of K , there are at least 2^d subsets ofC, so 2^d ∞ |C|. Consequently,VCD_(C)∞log₂|C|. This shows that ifCis finite, then VCD(C)is finite. As we shall see, the converse is false: there exist infinite collections Cthat have a finite V C-dimension.

If U is a finite set, then the trace of a collectionC= {C₁, . . . ,C_p}of subsets of U on a subset K of U can be presented in an intuitive, tabular form. Suppose, for example, that U = {u₁, . . . ,u_n}, and letλ=(T_C,u₁u₂· · ·u_n,r)be a table, where r=(t₁, . . . ,t_p). The domain of each of the attributes u_i is the set{0,1}.

Each tuple t_kcorresponds to a set C_kofCand is defined by

t_k[u_i] =

1 if ui ∪Ck, 0 otherwise,

for 1∞ i ∞n. Then,Cshatters K if the content of the projection r[K]consists of 2^|K|distinct rows.

Example 3.53 Let U = {u1,u2,u3,u4}and letCbe the collection of subsets of U given by

C= {{u2,u3},{u1,u3,u4},{u2,u4},{u1,u2},{u2,u3,u4}}. The tabular representation ofCis

126 3 Combinatorics

T_C u₁u₂u₃u₄ 0 1 1 0 1 0 1 1 0 1 0 1 1 1 0 0 0 1 1 1

The set K = {u₁,u₃}is shattered by the collectionCbecause r[K] =((0,1), (1,1), (0,0), (1,0), (0,1))

contains the all four necessary tuples(0,1),(1,1),(0,0), and(1,0). On the other hand, it is clear that no subset K of U that contains at least three elements can be shattered byCbecause this would require r[K]to contain at least eight tuples. Thus, VCD_(C)=2.

Every collection of sets shatters the empty set. Also, ifCshatters a set of size n, then it shatters a set of size p, where p∞n.

For a collection of setsCand for m ∪ N, letΨ_C[m]be the largest number of distinct subsets of a set having m elements that can be obtained as intersections of the set with members ofC, that is,

Ψ_C[m] =max{|CK| | |K| =m}.

We haveΨ_C[m] ∞2^m; however, ifCshatters a set of size m, thenΨ_C[m] =2^m. Definition 3.54 A Vapnik-Chervonenkis class (or a VC class) is a collectionCof sets such thatVCD_(C)is finite.

Example 3.55 LetRbe the set of real numbers and letSbe the collection of sets {(−√,t) | t ∪R}. We claim that any singleton is shattered byS. Indeed, if S= {x} is a singleton, thenP({x})= {∅,{x}}. Thus, if t x, we have(−√,t)∩S= {x}; also, if t<x, we have(−√,t)∩S= ∅, soSS=P(S).

There is no set S with|S| = 2 that can be shattered byS. Indeed, suppose that S = {x,y}, where x < y. Then, any member of S that contains y includes the entire set S, soSS = {∅,{x},{x,y}} ∅=P(S). This shows thatSis a VC class and VCD_(S)=1.

Example 3.56 Consider the collectionI = {[a,b] | a,b ∪ R,a ∞ b}of closed intervals. We claim thatVCD_(I)=2. To justify this claim, we need to show that there exists a set S = {x,y}such thatIS =P(S)and no three-element set can be shattered byI.

For the first part of the statement, consider the intersections

Fig. 3.2 Three-point sets can be shattered by half-planes

[u, v] ∩S= ∅, wherev <x, [x−π,^x+y₂ ] ∩S= {x}, [^x+y₂ ,y] ∩S= {y}, [x−π,y+π] ∩S= {x,y},

which show thatIS=P(S).

For the second part of the statement, let T = {x,y,z}be a set that contains three elements. Any interval that contains x and z also contains y, so it is impossible to obtain the set{x,z}as an intersection between an interval inIand the set T . Example 3.57 LetHbe the collection of closed half-planes inR², that is, the col-lection of sets of the form

{x=(x₁,x₂)∪R² | ax₁+bx₂−c0,a∅=0 or b∅=0}.

We claim thatVCD_(H)=3.

Let P,Q,R be three points inR²such that they are not located on the same line.

Each line in Fig.3.2is marked with the sets it defines; thus, it is clear that the family of half-planes shatters the set{P,Q,R}, soVCD_(H)is at least 3.

To complete the justification of the claim we need to show that no set that contains at least four points can be shattered byH.

Let{P,Q,R,S}be a set that contains four points such that no three points of this set are collinear. If S is located inside the triangle P,Q,R, then every half-plane that contains P,Q,R also contains S, so it is impossible to separate the subset{P,Q,R}. Thus, we may assume that no point is inside the triangle formed by the remaining three points (see Fig.3.3). Observe that any half-plane that contains two diagonally opposite points, for example, P and R, contains either Q or S, which shows that it is impossible to separate the set{P,R}. Thus, no set that contains four points may be shattered byH, soVCD(H)=3.

128 3 Combinatorics

Fig. 3.3 A four-point set cannot be shattered by half-planes

Fig. 3.4 Rectangle that separates the set{P_n,P_s,P_e}

Example 3.58 LetR²be equipped with a system of coordinates and letRbe the set of rectangles whose sides are parallel with the axes x and y. Each such rectangle has the form[x0,x1] × [y0,y1].

There is a set S with|S| =4 that is shattered byR. Indeed, let S be a set of four points inR²that contains a unique “northernmost point” P_n, a unique “southernmost point” P_s, a unique “easternmost point” P_e, and a unique “westernmost point” P_w. If L ∧S and L ∅= ∅, let R_L be the smallest rectangle that contains L. For example, we show the rectangle R_L for the set{P_n,P_s,P_e}in Fig.3.4.

On the other hand, this collection cannot shatter a set of points that contains at least five points. Indeed, let S be a set of points such that|S|5 and, as before, let P_nbe the northernmost point, etc. If the set contains more than one “northernmost”

point, then we select exactly one to be P_n. Then, the rectangle that contains the set K = {P_n,P_e,P_s,P_w}contains the entire set S, which shows the impossibility of separating the set K.