Norms and normed vector spaces

When dealing with vector spaces it is common to talk about the length and direction of the vector, and there is an intuitive geometric concept as to what the length and direction are.

There are a variety of ways of defining the length of a vector. The mathematical concept associated with the length of a vector is the norm, which is discussed in this section. In

94 Signal Spaces

section 2.4 we introduce the concept of an inner product, which is used to provide an interpretation of angle between vectors, and hence d~rection.

Definition 2.22 Let S be a vector space with elements x. A real-valued function llxll is said to be a norm if iixll satisfies the following properties.

N1 llxll

>-

0 for any x E

S.

N2 jjxlj = 0 if and only if x = 0.

N3 licrxli = lcrl ljxll, where cr is an arbitrary scalar.

N4 llx

+

y

ll

5 llxll

+ ll

y

11

(triangle inequality).

The real number l!xll is said to be the norm of x, or the length of x. ⁰ The triangle inequality N4 can be interpreted geometricaIly using figure 2.9, where x, y, and z are the sides of a triangle.

Figure 2.9: A triangle inequality interpretation

A norm "feels" a lot like a metric, but actually requires more structure than a metric.

For example, the definition of a norm requires that addltion x

+

^y and scalar multiplication crx are defined, which was not the case for a metric.

Nevertheless, because of their similar properties, norms and lnetrics can be defined in terms of each other. For example, if llxli is a norm, then

is a metric. The triangle inequality for metrics is established by noting that

(This trick of adding and subtracting the quantity to make the answer come out right is often used in analysis.) Alternatively, given a metrlc d defined on a vector space, a norm can be written as

llxll = d ( x , 01,

the distance that x is from the origin of the vector space.

Example 2.3.1 Based upon the rnetrlcs we have already seen, we can readily define some useful norms for n-dimens~onal vectors

1. The 1, norm: lix/il =

x:=,

^{ir, 1.}

2. The l2 norm: /ixl/, = (x:=, ].xi

l p )

^I".

3. The 1, norm: //x//, = Inax,,l,z, ,,, ^/.x, I

Each of these nonns introduce5 iti own geometry Cons~der. for example. the unit '"sphere" defined by

S, = ( x E R' / / x / / , ~ 5 I )

Figure 2 10 1llu5tratei the \hape of ~ c h \phere\ for various \ d u e \ of (7

-

' 3 Norms and Normed Vector Spaces

-.-

95

Figure 2.10: Unit spheres in

W2

under various 1, norms

Example 2.3.2 We can also define norms of functions defined over the interval [ a , b]

1. The L I norm: Ilx(t)lli

=So

b ^lx(t)ldt.

1 IP

2 The L , norm: l/x(t)l/,, =

(J"~

1x(t)lpdt) for 1 5 I ) < 3~

3. The L, norm: llx(t)jj, = suptci, hl Ix(t)l.

The 1, and L , norms are referred to as the uniform norms.

Definition 2.23 A normed linear space is a pair (S,

//

.

I/),

where S is a vector space and

I/

^.

/I

is a norm defined on S. A normed linear space is often denoted simply by S . When discussing the metrical properties of a normed linear space, the metric is defined in terms of the norm, d (x, y) = / / x - y

/ I .

Definition 2.24 A vector x is said to be normalized if /[xll = 1. It is possible to normalize any vector except the zero vector: y = x//lxll has ilyll = 1. A normalized vector is also

referred to as a unit vector. 0

With a variety of norms to choose from, it is natural to address the issue of which norm should be used in a particular case. Often the 12 or L2 norm is used, for reasons which become clear subsequently. However, occasions may arise in which other norms or norm- like functions are used. For example, in a high-speed signal-processing algorithm, it may be necessary to use the 1, norm, since it may be easier in the available hardware to compute an absolute value than to compute a square. Or, in a problem of data representation of audio information (quantization), it may be appropriate to use a norm for which a representation is chosen that is best as perceived by human listeners. Ideally, a norm that measured exactly the distortion perceived by the human ear would be desired in such an application. (This is only approximately achievable, since it depends upon so many psychoacoustic effects, of which only a few are understood.) Similar comments could be made regarding norms for video coding. In short, the norm should be chosen that is best suited to the particular application.

The exact norm values computed for a vector x change depending on the particular norm used, but a vector that is small with respect to one norm is also small with respect to another norm. Norms are thus equivalent in the sense described in the following theorem.

96 Signal Spaces

Theorem 2.3 (Nor171 equ~valence fheorer71) If

//

.

11

^and

I/

.

11'

are two rzorms or? .TRY (or

en),

then

/Ixn//-+O a s k i i x ; ifandorzlyif llxr1l1+O a s k i i x ; .

The proof of this theorem makes use of the Cauchy-Schwarz inequal~ty, w h ~ c h is introduced In sectlon 2.6. You may want to come back to thls proof after readlng that sect~on.

Proof It suffices to show that there are constants cl. c-, > 0 such that

To prove (2.1 l ) , it suffices to assume that

//

.

/I'

^{is the} l2 norm. To see this, observe that if d i lixll F llxll:! F d:!llxll and d; 11x11' 5 llxllz 5 d;llxll'

then

and

so (2.1 1) holds with cI = d l /dl and c-, = d2/d;. Let x be expressed as a linear coinbination of basis vectors

n

Then. by the properties of the norm,

The sum on the right is simply the inner product of the vector composed of the magnitudes of the xi's with the vector coinposed of the magnitudes of the basis vectors. Being an inner product, the Cauchy-Schwarz inequality applies. and

Let

Then the left inequality of (2.1 I ) applies with cl = 1/B.

For points x on the unit sphere S = {x: Ilxl12 = 11, the norm

//

.

/I

must be greater than 0 (by the properties of norms) and, hence, ljx!i 2 a for some a > 0 for x E S. Then

so the right-hand inequality holds with c~ = l / a . For example,

llxll2 5 l l ~ l ! l 5 fillxll2.

Ilxllx 5 !lxll7 5 f i I l x ! l x . llxll% 5 llxIl1 5 nllxllx.

1 r, Inner Products and Inner-Product Spaces 97

~efinition 2.25 For a sequence (x,}, if there exists a number M < cx such that

then the sequence is said to be bounded. 0

2.3.1 Finite-dimensional normed linear spaces

7 he notlon of a closed set and d complete set were introduced In sectlon 2 1 2 As po~nted havlng complete sets is advantageous because all Cauchy sequences converge, so that convergence of a sequence can be established s ~ m p l y by determining whether a sequence is Cauchy

Fln~te-dimensional normed Ilnear spaces have several very useful properties 1. Every finite-dimensional subspace of a vector space is closed.

2. Every finite-dimensional subspace of a vector space is complete.

3. If L: X -+ Y is a linear operator and X is a finite dimensional normed vector space, then L is continuous. (This is true even if Y is not finite dimensional.) As we shall see in chapter 4, this means that the operator is also bounded.

4. As observed above, different norms are equivalent on Rn or

en.

In fact, in any finite dimensional space, any two norms are equivalent.

A lot of the issues over which a mathematician would fret entirely disappear in finite- dimensional spaces. This is particularly useful, since many of the problems of interest in signal processing are finite dimensional.

We will not prove these useful facts here. Interested readers should consult, for example, 1238, section 5.101.

Dans le document Methods and Algorithms for (Page 133-137)