On the storage of correlated patterns in Hopfield's model

HAL Id: jpa-00212374

https://hal.archives-ouvertes.fr/jpa-00212374

Submitted on 1 Jan 1990

HAL is a multi-disciplinary open access

archive for the deposit and dissemination of

sci-entific research documents, whether they are

pub-lished or not. The documents may come from

teaching and research institutions in France or

abroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire HAL, est

destinée au dépôt et à la diffusion de documents

scientifiques de niveau recherche, publiés ou non,

émanant des établissements d’enseignement et de

recherche français ou étrangers, des laboratoires

publics ou privés.

On the storage of correlated patterns in Hopfield’s model

J.F. Fontanari, W.K. Theumann

To cite this version:

On

the

_storage

of correlated

_patterns

in

_Hopfield’s

model

J. F. Fontanari

_{(1,* )}

and W. K. Theumann

₍₂₎

(1)

Division of

_Chemistry,

California Institute of

_Technology,

Pasadena, CA 91125, U.S.A

(2)

Instituto de Fisica, Universidade Federal do Rio Grande do Sul, Caixa Postal 15051, 91500 Porto

_Alegre,

_RS, Brazil

(Reçu

le 3 avril 1989, révisé le 18 octobre 1989,

accepté

le 21 novembre

₁₉₈₉₎

Résumé. 2014 On étudie les effets de

stockage de p

patterns

statistiquement indépendants

mais effectivement corrélés dans le modèle de

_Hopfield

de mémoire associative. Ceci conduit à proposer une

_{règle d’apprentissage}

locale

_qui,

en _augmentant les différences entre patterns,

conduit à une mémorisation d’efficacité

comparable

à celle des

_{règles d’apprentissage}

non locales. Abstract. 2014

The effects of

storing

p

statistically independent

but

effectively

correlated patterns in the

_Hopfield

model of associative memory are studied. This leads us to _propose a local

_learning

rule which

_{by enhancing}

the differences among the patterns allows the network to store them with

an

_{efficiency comparable}

to nonlocal

learning

rules. Classification

Physics

Abstracts 87.30G - 64.60C -

75. 10H

1. Introduction.

The statistical mechanics

_analysis

of

_feedback,

_fully

_connected,

neural networks has made

possible

to unveil a

_variety

of

_interesting

features of these

_systems

which would have been hard to detect

_{solely through}

numerical simulations

_{[1, 2].}

The most studied neural network is the

_Hopfield

model _{of associative memory : the} states of the neurons are

_{represented by Ising}

spins, Si =

+ 1

_{(active) or Si = -}

1

_(passive)

and the

system

of N

_interacting

neurons is

governed by

the Hamiltonian

_[3]

The stored

_{patterns {gr}

= _±

1 , £

=

1,

... , p }

are

_imprinted

on the

_Ji/s

_(synaptic

connec-tions)

by

the

_generalized

Hebb rule

(*)

Present address : Instituto de Fisica e Quimica de Sao

_Carlos,

Universidade de Sao Paulo, 13560 Sao Carlos SP Brazil.

The

_equilibrium

states are characterized

_by

an

_overlap

vector m of p

_components

defined

_by

i = 1

for g

=

1,

_{..., p.}

In this

_approach

the neural network is viewed as a

_system

of

_{Ising spins}

in contact with a

heat bath which simulates the

_{biological synaptic}

noise. The tools

_developed

_{for infinite range}

spin-glasses

[4]

allow an

_{analytical study}

of the

_equilibrium

_properties

of neural networks

[1, 2].

It has been shown that there exist so called mixture states, i. e . states that are linear

combinations of the stored

_patterns,

in which m has several

_macroscopic,

_0(1),

_components

[1].

That

_study

_uncovered,

somewhat

_{surprisingly,}

that the

_synaptic

noise _{suppresses the}

mixture states in favour of the retrieval states in which m has

_only

one

_macroscopic

component.

Storing

correlated

_patterns

in the

_Hopfield

model is a

_problem

that involves mixture states since the network’s state must have either a non-zero

_overlap

with all the

_patterns

or with none. _{In this paper} we consider the

_symmetric

and

_asymmetric

mixture states. The former describes the situation in which the network confuses the

_patterns,

i.e. it cannot

_perceive

the individual details which make a

_pattern

different from the others. This state is described

_by

the vector

N

where

_mp

= j7 iE et’ Si

for JI- =

1,

_{..., p. The}

asymmetric

_{state we}

consider,

for

simplicity,

N ;

= 1

distinguishes only,

one of the

_pattems

from the other p -

_1,

and is described

by

the vector

where and for

These two states

_capture

the essence of the

_problem

of

_storing

correlated

_patterns

in

Hopfield’s

model : the network’s

_tendency

to enhance the common

_part

of the

_patterns

makes it more difficult to retrieve the details that

_distinguish

them.

Our

_study

is restricted to a

_particular

_type

of correlations in which the

_patterns

are

statistically independent

random variables

_{generated by}

the

_asymmetric

distribution

with

_bl

=

(1

+

_{a )/2,}

_b2

=

(1 -

a )/2,

and a E

_{[- 1, 1 ].}

_{Though independent,}

the

pattems

are

effectively

correlated since

for IL ::1=

v . Due to this

_particular

choice of correlations the

_asymmetric

state _{is p}

_{degenerate ;}

any

component

we choose to _{be ml} leads to an

_{equivalent asymmetric}

state.

This paper is

organized

as follows. In section 2 _,we

_study

the

thermodynamics

of the

Hopfield

model in the

_regime

_{p /N -}

0 when Nu oo. The

_simplicity

of the model

singles

out

the effects of the correlations among the

patterns.

In section 3 we _propose a local

_learning

which enhances the

asymmetric

state

_{by suppressing}

its

_rival,

the

_symmetric

state. The network’s overall

_performance

is

_comparable

to the nonlocal modified Hebb rule

_{proposed by}

Amit et al.

_[5]

where the

_nonlocality

is due to the

_{parameter a}

which _{stands for the average}

_activity

rate of the entire network. In section 4 we discuss our results and

_present

some

_concluding

remarks.

2. The

generalized

Hebb rule.

The

_design

of _{associative memory models may be}

_thought

of as an

_{optimization problem}

[6].

Given p

_{patterns { gr, IL}

=

1, ..., p}

to be stored in the network we must choose the connection matrix such that the local minima of the cost function or _energy occur when the network’s state

_S1

is near each one of the

_patterns.

The

_simplest

_{guess is}

where m’ is defined

_by

_{equation (1.3).}

Notice

that,

except

for the

_diagonal

term, this

equation

is identical to

_equation

_(1.1).

A rather different criterium for the

_design

of an

effective associative memory

model,

which we do not _{pursue in} this _{paper, is} that the basins of attraction of the local minima be as

_large

as

_possible

[7],

and some _{progress in this direction}

has

_already

been made with correlated

_patterns

_[8].

The

_{thermodynamics}

of the Hamiltonian

_(2.1)

has been

_fully

studied

_by

Amit et al. when the stored

patterns

are uncorrelated

_{[1, 2].}

This section focuses on the

_problems

of

_using

the

generalized

Hebb

_rule,

_equation

_(1.2),

to store correlated

patterns.

Since the

_diagonal

term in

_(2.1)

does not affect the

_{thermodynamics,}

we can

straightfor-wardly

take the average free _energy

_density

from reference

_[5],

p

where

_{m. e = E}

M »

e,"

and m," is

_{given by}

the saddle

_point

_equation

W=1 1

for g

=

1,

..., p. The

site subscripts

were

_dropped

since the

_{self-averaging}

_property

of 1 N

f

allows us to

replace N

_{(... )}

_by

_{the averages}

_>

over the

e’s,

and

_{a -1-}

T is a

N i =1

i

parameter

measuring

the amount of noise

_acting

in the network. Next we consider two

particular

solutions of

_equation

_(2.3).

z

2.1 SYMMETRIC SOLUTIONS. -

where and stands for

with and

Expanding

equation

(2.4)

in powers of

_mp

one finds

where

and the critical

temperature

Tc

at which the

_symmetric

state

_undergoes

a continuous transition to the

_paramagnetic

state

_(mP

=

0 )

is

This

_equation

is an indication that the correlations among the

_patterns

make the

_symmetric

state more robust to noise effects. However this transition

_only

occurs if the

_symmetric

solutions are stable near

_T,,

which is not the case for a =

0,

since the

odd-p symmetric

solutions are unstable above a certain

_temperature

0

_Tp

1 _{and the even-p} are

_always

unstable

_[1].

Next we show how these results

_change

_forez

0.

The elements of the matrix A whose

_eigenvalues

determine the local

_stability

of the saddle

point

solutions,

equation

(2.3),

are

which for the

_symmetric

solutions are reduced to

where

There are two

_types

of

_{eigenvalues :}

. a

_{nondegenerate}

_eigenvalue

. _{a p - 1}

_{degenerate eigenvalue}

The

_signs

of these

_eigenvalues

determine the local

_stability

of the

_symmetric

solutions. In the limit T -> 0 one

_{finds q =}

1 -

_prob

_(z.

=

0)

and

Q .- a 2 - prob

(z.

= 0 ).

Since

prob

_(z.

=

0)

=

Cp,p/z(b1 b2 )p/2

is nonzero

_only

_{for even p, the}

_odd-p

_symmetric

solutions

Near

_Tc

the

_expansion

of

_equations

_(2.lla-b)

_{in powers of}

_mp

_gives

and

Substituting

these results into the

_equations

_fors

1

and À 2

yields

Hence

1 is

always

positive

below

_Tc

_{and À 2}

becomes

_negative,

_signaling

the

_instability

of the

symmetric

solutions,

only

in the limit

a2 -+

0 for T --

_T,.

For non-zero a, and T not too far

below

_{Tc, À 2}

is

_positive

_{independently}

of the

_parity

_{of p.}

This

_implies

that the even-p solutions must become stable above a certain 0

_Tp

T,,.

To

_identify

the

_regions

of

_stability

of the

_symmetric

_{solutions in the space of}

_parameters

a and T we

_perform

a numerical

_analysis

_{Of À 2, equation (2.13),}

since it is the first

_eigenvalue

to

_change

of

_sign.

The results for several values of

_{odd p are presented}

in

_figure

_1,

where the

odd-p

solutions are unstable inside the contours of

_{Tp (a ).}

_Increasing

_{p increases the}

_region

of

stability.

For each p there is a critical value of a above which these solutions are stable for all

T

_T,.

The results for even-p are

_presented

in

_figure

_2,

where the even-p

_symmetric

solutions

become stable above

_T.

_T,.

This

_figure

shows

_clearly

that the role

_{played by}

the

_synaptic

noise

_changes

when the

_patterns

are correlated - it stabilizes the

_symmetric

mixture state.

So far we have studied the network’s

_tendency

to enhance the common information

contained in the

_patterns

reflected in the

_{increasing stability}

of the

_symmetric

state. Next we

show how this

_{tendency jeopardizes}

the network’s

_ability

to retrieve individual details of the stored

patterns.

2.2 ASYMMETRIC SOLUTIONS. - The

asymmetric

solutions,

equation

(1.5),

obey

the

equations

p

where S, = ml -t mp - 1 zp - 1,

and

_{zp _ 1}

_{= E e," .}

For weak

_{correlations,}

i.e. a « 1 these

/t=2

Fig. 1.

Fig.

2.

Fig.

1. - The

odd-p symmetric

solutions are unstable inside the contours of

_T.

_{(solid curves)}

shown for p = 3, 5, 7, 9, 21 and above

T,,

(broken curves)

shown for p = 3 and 21.

Fig.

2. -

The even-p

symmetric

solutions are unstable below

_Tp

_{(solid curves)}

shown for p = 2, 4, 6, 12 and above

7c (broken curves)

shown

for p

= 2 and 12.

Thus

_{making a}

= 0 one recovers the Mattis solutions. Notice that the retrieval

_overlap

ml is

only

slightly

reduced

by

the presence of weak correlations among the

patterns.

Next we consider the zero

_temperature

solutions of

_equations

_(2.17).

_Taking

the limit

f3 -+

00 in

_equations

_(2.18)

one finds

which

_clearly

is the best the network can do for

_retrieving

_patterns

correlated

_according

to

equation

(1.7).

However,

it can be

_easily

verified that this solution exists

_only

if

S’+

and E -

are

_positive.

This condition is satisfied for a _{a,, where}

The behaviour for a > _a,,

_depends

on the

_parity

of p. For

_{odd p}

the model

undergoes

a

where k’-=

_[p/2]

stands for the

_integer

_part

_{of p/2 [9].}

Since for even _{p the}

symmetric

solutions are

unstable,

the model must

_undergo

a discontinuous transition to another

asymmetric

state with

This solution has

_{mP _ 1 ml 1}

and for

_{large p}

tends to a

_symmetric

solution with

ml = mp - 1 = a.

It should be

_emphasized

that

_equations

_{(2.19), (2.21),}

and

_(2.22)

are not the

_only

zero

temperature

solutions of

_{equations (2.17) [10]}

but

_they

are the relevant ones for our _purpose

since their basins of attraction contain the

_input

states such _{that ml .-; 1 and}

_{mp _}

1 eue 0 which

bias the network to

_distinguish

_pattern

1 from the

_{other p -}

1. We also remark that the transition

_occurring

at a _{= ac is} not a

_{thermodynamic}

transition because the energy of the states

_plays

no role in the transition.

_{Nevertheless,}

_a,

_{clearly signals}

a

_change

in the

dynamical

behaviour of the model which is the

_important

information for the software and hardware

implementations.

To describe the behaviour of the model for T > 0 we associate the solutions

(2.19), (2.21),

and

_(2.22)

with the

_phases

_asymmetric

I

_(AI),

_symmetric

_(S),

and

_asymmetric

II

_(AII),

respectively.

The solution _ml =

mp _

= 0

corresponds

to the

paramagnetic

phase

(P).

For

odd p

the

_phase

_diagram

has two transitions : a discontinuous one from

_phase

AI to

_phase

S and a continuous one from

_phase

S to

_phase

P

_{given by}

_{equation (2.8).}

_Figure

3a illustrates

these transitions

_{for p}

= 3. In the _case of even p _we need to include the

phase

AII. This

phase

is very sensitive to noise and

_undergoes

a continuous transition to

_phase

S as soon as the

symmetric

solutions become

stable,

i.e. when

_a2

= 0. The

remaining

transitions are similar to the

_odd-p

case.

_Figure

3b shows the

_phase

_diagram

_{for p}

= 4. Notice the robustness of the

phase

AI to noise and the

_large

domain of the

_phase

S in both

_diagrams.

3. A

learning

rule for correlated

patterns.

An effective

_leaming

rule for

_storing

correlated

_patterns

should enhance their dissimilarities

or, which is the same,

penalize

their similarities. To

_implement

this idea we must

_design

a

network whose cost function or _{energy is maximized}

_by

the

_symmetric

mixture state. A

_simple

guess is

where m" -

_{m v 1ft}

_{and p}

_{= p - 1 with} m’ defined

_{by equation}

_(1.3).

A less

_transparent

_expression

for

H

can be obtained

_{by expanding}

the

_quadratic

term in

Fig.

3. - Phase

_diagram

for

_{(a) p}

= 3 and

(b)

_p = 4. The broken _curves

correspond

to continuous transitions while the solid curves

_correspond

to discontinuous transitions.

where

_Ji,

is a local

_learning

rule

_{given by}

Notice that

_going

from

_equations

_(3.1)

to

_(3.2)

we have omitted the

_diagonal

term

lii

since it

_plays

no role in the statistical mechanics

_analysis

of the model.

_However,

it does

affect the

_{dynamical properties}

in a nontrivial way

_[11]

_and,

to avoid future

_ambiguities,

we

define the model

_{by equations}

_(3.2)-(3.3).

The

_{thermodynamics}

of the Hamiltonian

_(3.1)

or

_(3.2)

is

_{straightforward.}

The

_partition

Deforming

the contours of

_integration

so that

_they

_{pass the} saddle

_point

and

_using

the

self-averaging

property

of the

_averaged

_{free energy}

_density

one finds

where m’ and t "

_obey

the saddle

_point

_equations

Clearly

the

_symmetric

state,

equation (1.4),

is not a solution of these

_equations.

This is the main consequence of our

_learning

rule

_being

different from that of Amit et

al.,

_equation

(1.9),

which has an _energy

_landscape

dominated

_{by spurious symmetric}

states. We retum to a

comparison

of the two rules below.

For the

_asymmetric

solutions,

equation

(1.5),

the saddle

_{point equations}

become

After

_averaging

over e 1,

_introducing

the variable

_{M = m 1 - m p _ 1,}

and

_eliminating

t1

and

tp -

1 one

gets

the

following

equations

where

r

We now consider the zero

_temperature

solutions of these

_{equations. Taking}

the limit

{3 -+

oo in

_equation

_(3.10b)

and

_multiplying

both sides

_by

M one finds

since

1 --t zp - i-- 0.

_Tuming

to

_equation

_(3.10a),

we notice that

_e+

and

0-

are

_always

p

positive

except

for

_{zp - 1}

= -

(p - 1)

and

zp -

1 = p - 1, respectively,

where

they

vanish.

Hence

_taking

the limit

8 --+

oo the

_only

solutions of

_équations

_(3.10)

are

which

_rapidly

_{approach (2.19)}

as p increases. These

equations give

us a clue to

_{understanding}

how the network works. Take a certain bit in

_pattern

{},

_say

ei.

The

_probability

of

e ’

= e i

for g =

2,

..., p is

r = bP

+

_bf

since the

patterns

are

_independent.

Thus the mean

number of bits which are common to all the

_patterns

is N T . Because the

_learning

rule

penalizes

the

_symmetric

state the network’s state must be such that half of these N T bits are reversed

_(if

all them are reversed the network would in fact enhance the

symmetric

state due to the

_{symmetry e "}

_{--+ - e e)}

and the N -

_{N T /2}

_remaining

ones are

equal

to the

_eils.

Hence

-in

_agreement

with

_equation

_(3.13a).

If one thinks of the

_patterns

as

_pictures

then

NT is the common

_background

in the

_pictures

and what the network does is

_homogenize

the

background leaving

the

_principal

untouched.

It is

_interesting

to _compare our

_learning

_rule,

_{equation (1.8),}

which for

_{large p}

is

_essentially

with the rule of Amit et

_al.,

_equation

_(1.9).

_Noting

that

and

_neglecting

the fluctuations

_{of 0 (p-1/2),}

one recovers

_{equation (1.9).}

_{Nevertheless,}

these fluctuations are

_strong

_enough

to

_modify

_{the energy}

_landscape

_and,

_{consequently,}

the

thermodynamics

of the model. To illustrate

_this,

let us consider the T = 0 solutions of both

models in the limit of

_large

but finite _p.

For the rule

_(1.9)

the

_symmetric

solutions are

_{given by}

_[5]

while

for the

rule

_(1.8)

one has

_{mp = Hs = 0.}

The

_asymmetric

solutions are

_{ml = 1,}

mp _

1 =

a2

for both rules and possess the energy

Thus the role of

_{the O(P-1/2)}

fluctuations is to destabilize the

_symmetric

states. Notice that for rule

_(1.9)

the

_symmetric

_{solutions have lower energy} for a >

_{(1 - 2/or}

_)1/2,

_though

this serious drawback can be avoided

_{by imposing}

a

_global

constraint to the

_dynamics

_[5].

4. Discussion.

In _{this paper} we

_proposed

a local

_leaming

rule for

_storing

correlated

_patterns

in a neural network _{model of associative memory. The}

_learning

_rule,

_equation

_(1.8),

_emerges as a natural

result of

_posing

the

_design

of the neural network as an

_{optimization problem}

for the energy

function : find a

_learning

rule that enhances the

_patterns’

differences. Such a rule must

necessarily

be

_complex.

A

_simple

additive

_leaming

rule,

like the

_generalized

Hebb

_rule,

treats each

_pattern

as a new

_piece

of information to be stored even if the

_patterns

are correlated.

Instead,

equation

(1.8)

extracts and stores

_only

the new information contained in the new

pattern

presented

to the network. _{Since the process of}

_selecting

the new information involves

a

comparison

of the new

_pattern

with all the data

_already

stored in the network the

_learning

rule cannot be _{local in the space of the stored}

patterns.

A similar situation occurs in the

storage

of

_{hierarchically}

correlated

_patterns

where in order to store a

_pattern

(descendant)

the

network needs to recall information

_already

stored

_{(ancestor) [14,}

_15].

Our

_leaming

_{rule may}

have a

_promising

_application

in

_pattern

_{recognition problems}

where

_important

features of the

stored

_patterns

are hidden in the

_background.

We think that this

_point

deserves further

attention.

Throughout

this paper we have considered

_{only learning}

rules where all the

patterns

are

embedded in the network at once

_through

a

_prescription

for

_Jjj.

_Although

this

_unsupervised

learning

strategy

does not

_guarantee

the

_stability

of the

_patterns,

it allows an

_{analytical study}

of the retrieval process. It should be

emphasized

that,

in the context of

_{supervised learning}

rules,

there exist a local

_algorithm

- the

perceptron

algorithm

- which

can

_generate

the

appropriate synaptic

connections to stabilize a set of N correlated or uncorrelated

_patterns

[16].

It should be remarked that the results of section 2 could be attributed to the statistical

independence

of the biased

_patterns

_{e e e "> = e e> e ">}

= a 2

instead of to a true correlation effect. We believe this is not the case _{since those results agree with the} intuitive

expectation

that the correlations should favour the

_symmetric

mixture state and this

preference

should be enhanced in the presence of noise. Further evidence is

provided by

comparing

our _{results for p} = 2 with the results of reference

[13]

where

e,">

=

V)

= 0

_{and (}

_{e >}

= Q.

The

_equations

there are reduced to

_equations

_(2.17)

when one

replaces Q

by a2.

Among

the

_advantages

of the

_leaming

rule

_equation

_(1.8)

over

_equation

(1.9)

are the

absence of the

_symmetric

mixture state and the

_{applicability}

to _any set of correlated

_patterns

without earlier

_knowledge

of the correlations. To compare these two

_learning

rules in the

limit of non-zero a -

_p/N

we have run simulations for a = 0.1

(N

=

200 )

and measured the

retrieval

_overlap

_ml as a function of a. Rule

(1.9)

has

only

a retrieval

_overlap

Acknowledgments.

The research at Caltech was

_{supported by}

contract N00014-87-K-0377 from the Office of Naval Research. J. F. F. thanks the kind

_hospitality

of IF-UFRGS where this work was started. W. K. T. thanks R. Erichsen Jr. for aid with some of the calculations in section 2. The research

of W.K.T. was

_supported

in

_part

_by

Conselho Nacional de Desenvolvimento Cientifico e

Tecnologico

(CNPq)

and Financiadora de Estudos e

_Projetos

(FINEP),

Brazil. J.F.F. was

partly supported by

a

_CNPq

_fellowship.

References

[1]

AMIT D. J., GUTFREUND H. and SOMPOLINSKY H.,

Phys.

Rev. A 32

₍₁₉₈₅₎

1007.

[2]

AMIT D. J., GUTFREUND H. and SOMPOLINSKY H., Ann.

_Phys.

N. Y. 173

₍₁₉₈₇₎

30.

[3]

HOPFIELD J. J., Proc. Natl. Acad. Sci. USA 79

₍₁₉₈₂₎

2554.

[4]

BINDER K. and YOUNG P. A., Rev. Mod.

_Phys.

58

₍₁₉₈₆₎

801.

[5]

AMIT D. J., GUTFREUND H. and SOMPOLINSKY H.,

Phys.

Rev. A 35

₍₁₉₈₇₎

2293.

[6]

HOPFIELD J. J., and TANK D. W., Biol. Cibern. 52

₍₁₉₈₅₎

141.

[7]

BRUCE A. D., GARDNER E. and WALLACE D. J., J.

Phys. A

20

₍₁₉₈₇₎

2909.

[8]

GARDNER E., J.

Phys. A

21

₍₁₉₈₈₎

257.

[9]

See

Appendix

A of reference

_[1]

for a similar calculation.

[10]

THEUMANN W. K. and ERICHSEN R., Jr.,

unpublished.

[11]

FONTANARI J. F. and KOEBERLE R., J.

_{Phys. A}

21

₍₁₉₈₈₎

L667.

[12]

Throughout

this paper we have omitted the trivial solutions due to the _symmetry m ~ - m.

[13]

FONTANARI J. F. and KOEBERLE R. , J.

Phys.

A 21

₍₁₉₈₈₎

2477.

[14]

PARGA N. and VIRASORO M. A., J.

_Phys.

France 47

₍₁₉₈₆₎

1857.

[15]

FEIGELMAN M. V. and IOFFE L. B., Int. J. Mod.

_Phys.

B 1

₍₁₉₈₇₎

51.