Pattern recognition in Hopfield type networks with a finite range of connections

HAL Id: jpa-00212491

https://hal.archives-ouvertes.fr/jpa-00212491

Submitted on 1 Jan 1990

HAL is a multi-disciplinary open access

archive for the deposit and dissemination of

sci-entific research documents, whether they are

pub-lished or not. The documents may come from

teaching and research institutions in France or

abroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire HAL, est

destinée au dépôt et à la diffusion de documents

scientifiques de niveau recherche, publiés ou non,

émanant des établissements d’enseignement et de

recherche français ou étrangers, des laboratoires

publics ou privés.

Pattern recognition in Hopfield type networks with a

finite range of connections

Eva Koscielny-Bunde

To cite this version:

Short Communication

Pattern

_recognition

in

_Hopfield

_type

networks with

a

finite range

of

connections

Eva

_{Koscielny-Bunde(1)}

(1)

Fachbereich

_Informatik,

Universität

_Hamburg,

D-2000

_Hamburg

_50,

F.R.G.

(Received

on June

15,1990,

accepted

on June

20,1990)

Abstract. 2014 We

study

pattern

recognition

in linear

_Hopfield

type networks of N neurons where

each neuron is connected to the z

_subsequent

neurons such that the state of the

ith

neuron at time

t + 1 is determined

_by

the states of neurons i + 1,

...,i

+ z at time t. We find that for small values of

z/N

the retrieval behavior differs

_considerably

from the behavior of diluted

_Hopfield

networks. The maximum number of random

_patterns

that can be retrieved increases in a non linear _way with z and the

_asymptotic

mean

_overlap

between

_input

and

_{output patterns}

decreases

_sharply

as z is decreased and reaches zero at a finite value of z.

Classification

Physics

Abstracts 87.30 - 75. 10H - 64.60

In recent _years,

_Hopfield

neural networks

_[1]

have been studied

_extensively

_(for

recent

re-views see _e.g.

_[2-4]).

The

_Hopfield

network

_provides

a standard for neural nets since its

_long

time retrieval behavior could be treated

_analytically

_[5]

_by

_solving

for the

_equilibrium

statistical

me-chanics of the net. The solution could be achieved since all neurons in the net are interconnected.

In this

_paper

we

_report

on numerical studies of linear

_Hopfield

_type

nets of

_(usually

N =

400)

neurons where each neuron is connected

_only

to a certain number of _{other neurons, and}

_periodic

boundary

conditions are

_employed.

Previous work in this direction was

_mainly

devoted to nearest

neighbor

and next nearest

_neighbor

_connections,

or to dilute

_(damaged)

networks where a certain fraction of connections was

_randomly

cut

_[6-10].

The network consists of lV neurons, each of them can be in two states

_{?: = ±1.}

The

_coupling

strength Ji,j

between

_pairs

of connected neurons is determined

_by

the M random

_patterns

_{(g> )}

(eï

çfv), J-t

= 1,

2,...,

M,

which are to be stored in the network:

The state of neuron i at a

_given

time t

+ 1

is obtained from the states of the z

_subsequent

neurons

at time t

_by

-

1798

For z =

_{N - 1,}

₍₂₎

is identical to the retrieval

_process

in

_Hopfield

nets where each neuron receives

input signals

from all other neurons and sends

_output

_signals

to all other neurons. For z

N,

each neuron i receives

_input

_{signals only}

from the z

_subsequent

neurons i

_{+ 1,}

_...,i

+ z. The set of

neurons influenced

_by

neuron i and the set of neurons

_influencing

neuron i

_partially

_overlap

for

z >

N/2.

For z

_N/2,

the neurons in both sets are all différent.

A

_particular

_pattern

_te"}

is called stored

_by

the

network,

if an initial

_{configuration}

_{{5’(0)} ==}

(S1(0),

...,

8N(O))

near

le"}

develops

under the

dynamic

rule

(2)

to a state whose

overlap

with {ç I/}

is

_large.

The maximum number of

_{patterns Me}

that can be stored

_depends

on the size of the

_network,

the difference between

_input

and

_output

_patterns

_{(initial noise)}

and the number of connections

z

_per

neuron.

_According

to Amit et al.

_[5],

we have in the limit of

_{large Hopfield}

networks

(z = N - 1 - cxJ),

In this

_work,

we

_study

how

_{equation (4)}

is modified for z

_{N - 1,}

and how the retrieval rate

decays

in the nonretrieval

_regime.

In the numerical

_study,

first M random

_patterns

_lem}

are

_generated

and the connection

strengths J,j

are calculated. Then one of the nominated

_patterns

is chosen and a

_{noisy input}

pattern

is

_generated

where the state of each neuron is

_changed

with

_probability

_pn. The

_dynamic

rule

₍₂₎

is

_applied

and the time

_{dependent overlap}

_{qm (t)}

is calculated. The run

_stops

when

_{qm (t)}

stays

constant for six time

_steps

or oscillations of

_period

2 with constant

_amplitude

occur. This

procedure

is

_repeated

for

_typically

101

initial

_{configurations,}

which are chosen from different

nom-inated

_patterns.

_{By averaging}

the results we

_obtained,

for a

_given

number Il of

_patterns

and fixed initial noise _pn, the mean retrieval

_overlap

_q > as a function of z.

Figure

1 shows the

_dependence

of _q > on

_z/N,

for

_pure

_input

_patterns

_(pn

=

0)

and networks

with N =

120,

200 and 400 _neurons. In the different

networks,

the ratio

M/.N

has been

kept

constant at 0.075. One can

_distinguish

between 3

_{régimes : (1)}

the retrieval

_regime

for

_{large z}

where

_M/z

> 0.14 and _q > is close to _one;

₍₂₎

an intermediate transient

_regime

where _q >

is well below one but

_greater

than zero;

₍₃₎

a disordered

_régime

below some threshold value zc,

where _q > = 0 and the net behaves as if all connections were _{cut; Zc} seems to increase

_slightly

with

_increasing

size of the network. The width of the transient

_{region (2)}

shrinlcs if the size of the network is increased.

Figure

2

_shows,

_again

for

_pure

_input

_patterns,

_{the variation of q} > with the number of nominated

_{patterns M,}

for N = 400 fixed and three values of _{z: z} =

_399,

_240,

and

_{80; z}

= 399

corresponds

to the conventional

_Hopfield

net The result is

_plotted

versus

Mlz

rather than

_MIN.

For

_M/z

not too

_large,

the data

_{approximately collapse}

to a

_single

line. At intermediate values of

_M/z,

the data

_split

into three curves. For z =

N - 1,

a

_strong

decrease in the mean

_overlap

is followed

_by

some smooth behavior where _q > varies

_only

little with

M/z

and seems to reach a

plateau

> 0 at

_large

values of

_Mlz

_(see

also

_[5,11]

_).

For z =

240,

again

the curve

_{drops sharply}

and then decreases less

_strongly

with

_increasing

_M/z.

For z =

80,

finally,

the mean

_overlap

shows

a

_sharp

decrease with

_M/x

and reaches zero at

_finite

value of

_Mlz

_roughly

at

_{M/z ù}

0.3. We have

_compared

this behavior with the case where the connections between neurons have been

_{randomly destroyed}

_{[6, 10].}

In this case, z can be referred to as mean number of connections

per

neuron which is related to the concentration _p of

_damaged

bonds

_by

z =

(1- p)N.

In

figure

2,

Fig.

1. - Plot of the

asymptotic

mean

_overlap

as a function of

_z/N

for N =

_{120( Ó),}

_200(0),

_and

_400(o).

The number of stored _{patterns per} neuron is

_kept

fixed,

M/N

= _0.075, and pn = _0.

Fig.

2. - Plot of the

_asymptotic

mean

_overlap

as a function of the number of nominated

_patterns

_per

con-nection,

M/z,

for N = ₄₀₀ and _pn = _0. Different

_symbols

represent

different z- values : z =

_80(A),

z =

_240(0),

_{and z} =

399(o).

The results for the

damaged

networks are indicted

_by

A

_{(corresponding}

to

z =

_80),

and

_bye

(corresponding

to z =

240).

and _p =

0.8,

_{respectively.}

The curves differ

strongly

from the

corresponding

results for

z/N

= 0.6

and 0.2. In the diluted

_network,

the mean retrieval

_overlap

does not scale with

_M/z

and varies

comparatively weakly

with

_M/z.

Figure

3

_finally

shows the maximum number of

_patterns

_per

_connecting

bond

_Mlz

that can

1800

pn. For

pure

input

patterns

(Pn = 0)

and

_z/N

>

_1/4,

_Mc/z

is

_{approximately}

the same as for the

_Hopfield

net,

which also

_agrees

with the situation in diluted

_Hopfield

nets

_{[10] .}

Below

_{zIN c--- 1/4,}

_Mc(z)/z

decreases almost

_linearly

with

_z/N,

with the

_{exception Mc /z}

= 1 for M = _{1. Almost the same}

curve is obtained for _{p, =} 0.2

_except

for M = 1 and at

_large

values of

_z/N

where

_Mc/z

is

considerably

smaller than for

_pure

_input

_patterns.

For _very

_{noisy input}

_patterns

_{(p, = 0.4),}

the

curve

_again

is rather flat and

_except

at small values of

_z/N

_depends

rather

_weakly

on z. It is

interesting

to note that in all cases

_{Mc /z}

does not _{show any}

_pronounced

features around the "critical" value

_z/N

=

1/2

where the sets of

_input

and

_output

neurons start to have no

_overlap.

The behavior shown in

_figure

3 differs

_clearly

from the behavior in diluted networks where

_{Mc/ z}

is described

_by

a

_simple

crossover behavior. Below some crossover value zr

(pn )

which decreases

with

_increasing

noise level _pn,

_Mc/z

is

_roughly

constant and follows

_(5),

while above

_{zr mc /z}

bends down

_[10].

In

_particular

for

_large

_noise,

_Mlz

decreases as

_1/z

for z

_approaching

one,

which is _very différent from the behavior observed here.

Fig.

3. - Maximum fraction

Mc/ z

of learned random

_patterns

in a network of 400 neurons that are

rec-ognized

with less than 0.5 _{percent error,} as a function of the number of connections per neuron _z, for the initial noise _pn =

0(a), 0.2(o),

and

0.4(A).

It is a

_pleasure

for me to thank

_{Greg Kohring}

and Dietrich Stauffer for valuable discussions and critical

_reading

of the

_manuscript.

References

[1]

HOPFIELD

J.J.,

Proc. Natl. Acad. Sci. USA 79

₍₁₉₈₂₎

2554.

[2]

ECKMILLER

_R.,

and v. MAISBURG Ch.

_eds.,

Neural

_Computers

_(Springer

_Verlag,

Berlin,

_Heidelberg)

1988;

SOMPOLINSKY

_H.,

_{Physics Today}

41

₍₁₉₈₈₎

70.

[3]

DOMANY

_{E., J.}

Stat.

_Phys.

51

₍₁₉₈₈₎

743.

[4]

AMIT

D.J.,

Modelling

brain function: The world of attractor neural networks

_{(Cambridge University}

[5]

AMIT

D.J.,

GUTFREUND H. and SOMPOLINSKY

_{H., Ann.}

_{Phys. 173 (1987)}

30.

[6]

DERRIDA

B.,

GARDNER E. and ZIPPELIUS

_A.,

_Europhys.

Lett. 4

₍₁₉₈₇₎

_167;

CANNING A. and GARDNER

E., J.

Phys.

A 21

₍₁₉₈₈₎

3275.

[7]

KÜRTEN

_K.E.,

J.

_Phys.

France 50

₍₁₉₈₉₎

2313,

and in _press.

[8]

FORREST

B.M., J.

Phys.

A 21

₍₁₉₈₈₎

245 and J.

_Phys.

France 50

₍₁₉₈₉₎

2003.

[9]

NOEST

_A.,

_Phys.

Rev. Lett. 63

₍₁₉₈₉₎

1739.

[10]

CLARK

J.W.,

RAFELSKI J. and WINSTEN

_{J.V., Phys.}

_Rep.

123

₍₁₉₈₅₎

_205;

KOSCIELNY-BUNDE

E., J.

Stat.

Phys.

58

₍₁₉₉₀₎

1257 and in: Parallel

_Processing

in Neural

_Systems

and

_Computers,

R.

Eckmiller,

G.

Hartmann,

G. Hauske Eds.

_(North

_Holland,

_Amsterdam)

_1990,

_p. 495.

[11]

KOHRING

G.A., J.

Stat.

_Phys.

59

₍₁₉₉₀₎

1097.