Simulation of a multineuron interaction model

HAL Id: jpa-00246462

https://hal.archives-ouvertes.fr/jpa-00246462

Submitted on 1 Jan 1992

HAL is a multi-disciplinary open access archive for the deposit and dissemination of sci- entific research documents, whether they are pub- lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Simulation of a multineuron interaction model

J. Arenzon, R. M. C. de Almeida, J. Iglesias, T. Penna, P. M. C. de Oliveira

To cite this version:

J. Arenzon, R. M. C. de Almeida, J. Iglesias, T. Penna, P. M. C. de Oliveira. Simulation of a multineuron interaction model. Journal de Physique I, EDP Sciences, 1992, 2 (1), pp.55-62.

�10.1051/jp1:1992122�. �jpa-00246462�

Classification

Physics

Abstracts

87.30G 64.60C 75.10H

Simulation of

_a

multineuron interaction model (*)

J. J. Arenzon

(I),

R. M. C. de Almeida

(I),

J. R.

Iglesias (I),

T. J. P.

Penna(2)

and P. M. C. de Oliveira

(2)

(1) Instituto de Fisica, Universidadc Federal do Rio Grande do Sul, C.P. 15051, 91500Porto

Alegre,

RS, Brazil

(2) ^Instituto de Fisica, Universidade Federal Fluminense, C.P. 100296, 24020 Niter6i, RJ, Brazil

(Received ²

July

1991,

accepted

in

final form

6

September1991)

Abstract We simulate the

dynamics

of _a muitineuron model (RS model) witli _an energy function

given by

tile

product

between the

squared

distances in

phase

_space between the state of tile net and the stored pattems. We obtain the relative

frequency f(m~)

that _an

arbitrary

pattem is retrieved from _an initial

overlap

_{m~ and} estimate the size of the basins of attraction for different

activities _a. Two limit cases are taken into account _: when pattems ^and

antipattems

_are stored

(p.a.s.)

and when

only

the pattems are stored

(o.p.s.).

For the _a

= 0.5, _p.a.s. nets a limit for the load parameter was not found, but for the other cases (a,0.5 or o.p.s.

configuration)

the

relative size of the basins of attraction may become too small.

Most _common models for neural networks consist of

systems

of _a great number of

interacting spins (neurons) [1, 2]

where each

configuration

is characterized

by

_an N- dimensional vector

S, given by

S

=

(Sj,

,

SN

)

;

S;

= ±

(1)

The Hamiitonian

function,

and hence the

configurations

that minimize the energy of the

system,

^is determined

by

the

spin

interactions considered

by

_a

given

model. For the attractor

neural

network,

when any set of P chosen states of the net

(pattems),

_say

(~,

~1 =

1,

...,

P,

_are

implemented

as minima of the Hamiltonian

through convenient,

_a

priori prescribed tuning

of the interactions

strengths,

the _net is said _to have leamt these

pattems.

^In this _{case, a}

given _pattem (~

is retrieved

when, starting

^from an

initial,

unstable

configuration

that is

sufficiently

similar to the pattem, ^{the net relaxes towards and stabilizes at the}

(minimum energy)

state

~~.

(*) Work

partially supported by

brazilian

agencies

Conselho Nacional de Desenvolvimento Cientifico

e

Tecno16gico (CNPq),

Financiadora de Estudos _e

Projetos

(FINEP), Coordenadoria de

Aperfeiqoa-

mento de Pessoal de Nivel

Superior

(CAPES) and

Fundaq6es

de Amparo h

Pesquisa

dos Estados do Rio

de Janeiro and Rio Grande do Sul (FAPERJ and FAPERGS).

56 JOURNAL DE

PHYSIQUE

I N° I

The _most

extensively

studied

example

of such idealized

systems

^{is the}

Hopfield model,

which in its

original

version considers the

following

Hamiltonian

[2]

_:

E

=

jj

_J,~

S,

S~

(2)

,,j i

and the

synaptic

matrix

J,~,

^{associated to the}

strengths

of

spin interactions,

is

given by

^Hebb

leaming

rule

[2, 3]

:

J,~ =

jj f,~f~~ (3)

N

~

with

(~,

_~1

=

1,

,

P, being

P _vectors in the

phase

_space

representing

the stored pattems.

This

synaptic

matrix J;~ guarantees ^that these

pattems

are

strongly

correlated _to minima of the Hamiitonian

equation (2),

^if

they

are uncorrelated among themselves and the load parameter

a, defined _as

a =

~

(4)

is less than _a critical value a~ = 0.14

[4].

In this limit _case the

Hopfield

model describes _a

content-addressable _memory. The uncorrelated pattems are chosen

by assigning equal

probability

_a

= 0.5 of each

spin if

to be _{+ or at} random. The value _a defines the

activity

of the _net and lower values of _a

(favoring spin +1,

for

instance)

introduce correlation between the stored memories. Several

attempts

have been made _to model _a neural network

capable

to handle

satisfactorily

with correlated

pattems

^{and with} a

higher

critical load

capacity,

here measured

by

_a~, ^both

by introducing

modifications in the

Hopfield

Hamiltonian

or

by proposing

new models

(see,

for

example,

Refs.

[4-11]).

Recently,

de Almeida and

Iglesias [12] proposed

a new Hamiltonian function that is related

to the distances in

phase

_space between the state of the _net S and the P pattems

(~.

This

model,

named RS after the brazilian _state where the model _was

created,

includes

multispin

interactions and _can be written

as

p

l~

^N

E

=

N

fl jj

_(S~

ff)~ (5)

~=i

~'~,=i

The

expression

in the

product

is the

squared

Euclidean distance between S and

(~, i.e.,

E is _a

non-negative

function and

E(S)

=

0 if S

=

(~,

for _any

~1=1,

.,

P.

Consequently

the

pattems

are

always

minima of the energy function. The

capacity

of the

network,

_as well _as its

ability

to handle with correlated

pattems

are

greatly

and

qualitatively

enhanced in

comparison

with

previous

models. Also, if

(~

is _a minimum of

equation (5),

it is

not a direct consequence that its

antipode -(~ (antipattem)

should also minimize the

Hamiltonian,

_as it

happens

in the

Hopfield

Model.

Nevertheless,

_{one can}

always explicitly

store the

antipattems,

if _one wishes _so. This feature of the model

yields

to two different

limiting

_cases for uncorrelated pattems : when

only

the pattems are stored

(o.p.s.)

_or when both

pattems

^and

antipattems

are stored

(p.a.s.).

The above considerations _were

directly

inferred from

equation (5)

and further conclusions

require deeper analysis

of the

dynamics implied by

the model. As the system ^under consideration takes into account _a great ^{number of}

interacting

_neurons, there _{are two} different

approaches

to the

problem

: mean field calculations

through

statistical mechanics

techniques,

which

preliminary

results have been

presented

in reference

[12],

and numerical simulations. In this work _we

present

some

computer

^{simulation results for the} two

limiting

cases

(o.p.s.

and p.a.s.

configurations)

where _we consider different load

parameters

^and activities for the random

pattems.

^{In what follows} we

briefly explain

the

procedures

to obtain the relative fraction

f (mo)

of times that _an

arbitrary _pattem

is retrieved when

starting

from _an

initial _state with _an

overlap

_mo,

following

the

prescription by

Forrest

[13].

We

analyze

the

results,

obtain _an estimate of the size of the basins of attraction and show that the load

capacity

of the net has been

greatly improved.

Consider _a network of N _neurons and _a zero-temperature

dynamics

such that _a

spin

is

flipped

whenever the energy of the _net is lowered.

Equation (5) gives

_a lot of information about the energy

landscape

in

phase

_space, but it is _not in _an

adequate

form for numerical

purposes. It _can be rewritten _as

p

E

= N

fl (i

_m~

) (6)

~=i

in the o.p.s. case, or

p

E

= N

fl (i m( (7)

~=i

in the p.a.s.

configuration [12].

In

equations (6)

and

(7)

_m~ stands for the

overlap

of the _state of the _net S with the

~1-th

_pattem :

m»

=

Z if St (8)

,

~

i

The numerical simulation is

performed

_as follows _:

I)

initialization _{: construct} P random

pattems

^with

activity

_a, and _an initial _state that has _a

given overlap

_mo with _one of the stored

pattems,

^chosen at random. Calculate the

overlaps

m~ ^{with the other} ones

through equation (8)

and then the initial energy

through (6)

_or

(7)

_;

it)

consider _a virtual

flip

in the I-th

spin

and calculate virtual

overlaps mf by

mi

= m»

~

S, if

_;

(9)

iii)

obtain the virtual energy E*

through equations (6)

_or

(7) using

the

overlaps

~ ^*.

» ,

iv)

whenever E* WE

flip

the

spin

and

update

the energy and

overlaps

v)

the

updating

is

performed sequentially

until the system ^reaches a stable state ; The steps

(I)

to

(v)

are

repeated

several times and the

averaged frequency f(mo)

that the

randomly

chosen

pattem

^{is retrieved from} an initial

overlap

_mo is obtained. We used the

multispin coding approach,

as

prescribed by

Penna and Oliveira

[14, 15]. Although

the number of arithmetic

operations

_per

updating

_are of the _same order _as in the

Hopfield model,

more

computing

time is

required

because here the energy function is calculated

through

the

product

of the

overlaps

instead of the _{sum as} in the

Hopfield-Hebb algorithm.

For _more

computing

details _see reference

[17].

Figure

shows the relative

frequency f(mo)

^{for the} p.a.s.

configuration

and 0.5

activity,

uncorrelated _sets of memories. The nets considered have N

=128,

256 and 512 and

simulations _were

performed

for _a

=

0.1,

0.5 and I. The average values

f(mo)

_were obtained

considering

200 relaxations

(50

for the N

= 512

net)

for each of five different sets of memories for every

point

of the

figure

and the estimated _{errors are} less than 10 9b. As the pattems are

always

minima of the energy

function,

when _a pattem ^{is retrieved the final}

58 JOURNAL DE PHYSIQUE I N°

1-O ~~ _j-~ ~ &@~

/ fi' _/

~ /

/O~

O /

O-B

/,~

/

/ /

/~~ ,

t°'~

Q~0. ~/ 0.5

~

~0.4

/

02 ~~11

~

~ ~ /°~

/~ _Ja ~/

00

0.0 0 2 0 4 0 6 0 fi 0

mo

Fig.

I. Fraction

f(mo)

of recalled pattems ^with an initial

overlap

_mo for N

= 128 (O), 256 (6) and

512 (a) and _« 0.1, 0.5 and I iri the p.a.s. case, and uncorrelated pattems.

overlap

is

always I, I.e.,

there is _{no errors} in the retrieved states.

Considering

the p.a.s. case,

similarly

to the

Hopfield model,

_curves

corresponding

to nets with the _same value of the load parameter a but different number of _neurons N

(and

hence different number P

= NtY of

stored

pattems)

_superpose

[13],

and the rate at which

f(mo)

increases form 0 to I becomes

more

pronounced

as the size of the system ^{N increases for all values of} a in the

figure.

As

suggested by

Forrest

[13]

this behavior

approaches

a

discontinuity

at

m~(a)

as N

~ cc

implying

that _even for

a =

i the _net

keeps

its

retrieving

abilities. The value lim m~

(a gives

N

~ w

a measure of the size of the basins of attraction of the memories.

Rough estimates,

taken

directly

from

figure I,

_are

m~(a

=

0.1)

₌

0.12, m~(a

=

0.5)

₌ 0.43 and

m~(a

=

1)

=

0.61.

Comparisons

with the

Hopfield

model value

mf(a

=

0.1)

= 0.37 shows

that,

_even at very low _a the size of the basins of attraction _are

greatly

enhanced in this model.

Also,

the

dependence

_{of m~ with} the load parameter a is much smoother than in

Hopfield

Model

(the

basins of attraction _{are not}

drastically reduced),

because here there is _no critical value for the load

parameter,

at least in the

region

_{a w} I. This effect could

originate

from the elimination of

spurious

states

by

the

higher

order

synaptic

connections considered

by

the Hamiltonian

equation (2). (At

^N _~ cc a few very shallow

spurious

states with Em 0 may appear in numerical simulations. At N

~ cc these states merge in _an absolute

(unstable)

maximum _{at a}

point

in

phase

_space that is

equidistant

from every

(uncorrelated)

stored pattem, ⁱⁿ agreement ^with mean field

(N

~ cc

)

calculations

[12, 16]). Higher

values of

a do not present any

qualitative

difference for the

analyzed

_case but for _a

expected

decrease in the size of the

basins of attraction.

In

figure 2,

_{we present}

f (mo)

versus mo for 0.5

activity,

uncorrelated

pattems

^{in the} o.p.s.

configuration

for _nets with N

=

128,

256 and 512 _neurons

again,

the average is taken _over

200 relaxations for each of five different sets of memories

(50

for the N

=

512).

Here the different N _curves with _same number P of stored pattems

(and

not

a)

_superpose. This _can be

explained

_as follows. The

antipattems

are located in _a

region

in

phase

_space that is the

specular

reflexion of the

corresponding region

where the

pattems

are. When antimemories

are also stored

(p.a.s. case)

these _two

regions

have similar energy

landscapes.

^{But when} antimemories _are not considered

(o.p.s. case),

the

antipattems region comprises

_very

large

energy states, while its

specular image

contains the lower energy ones,

forming

_an

analogue

of

a multi-hole bowl in the closed space defined

by

the

(hyper)

surface of the

(hyper)

cube in

1-o = ~Q

/

/ ~

o-B ~

/

/ /

d

-°.6

P=13 /~~~ ^{64 128}

~

o

_/~~

+-0.4 _/

~/

_l

~/

_l

O-Z

~

/

/ ~d

O-O

0.6 0.7 O-B 0.9 1-O

mo

Fig.

2.

f(m~)

_{versus m~} for N

=

128

(O),

256

(A)

and 512

(o)

and P

=

13, 64 and 128 in the o.p.s.

case, and uncorrelated pattems. ^{Note the different scale} on the horizontal axis.

phase

_space. The greater ^{the number of} pattems ^P

is,

the

bigger

the difference between these two

regions

and the

deeper

is the bowl. The

pattems

are stored inside this

big

basin whose center

lays

in the direction C determined

by

the _sum of all stored uncorrelated

pattems.

^{As P}

increases,

the sizes of the basins of attraction of each memory

decrease,

I.e. m~

decreases, enlarging

the

probability

that the net stabilizes in _a state which is

strongly

correlated _to the

center of the

big

bowl.

Figure

3 presents a

pictorial

sketch of the energy

landscape

in the

phase

_space for this _case. In the limit P

~ cc, the energy of the state in the center of the bowl goes ^to zero, the basins of attraction of each stored

pattem

merge, and there is

only

_one

big

basin with _a

flat,

_{zero energy} bottom in the

region

in

phase

_space which is delimited

by

_a

ring

obtained

by linking

the

points

associated to the pattems ^{and contains the direction C. Outside} this

ring

the energy increases

monotonically

from _zero to

infinity, reaching

its maximum value at the

point

C. Hence it is not

surprising

that in the o.p.s. case the relative size of the basins of

attraction,

measured

by

_m~, is determined

by

the absolute number P while for the p.a.s.

configuration

the relative

quantity

_{a =}

PIN,

that

gives

_{a measure} of the

density

of stored

pattems,

^{is the} relevant number in

defining

the size of the basins of attraction. This effect is in

agreement ^with mean field calculations

[16]. Again,

the rate at which

f(mo)

increases from 0

to becomes _more

pronounced

as the size of the system ^increases and _{we can see} that the net still retrieves the stored pattems ^{when P is of the order of}

N,

at least for finite _nets.

p<~~n p-on

~ B

AE

Iv f iv I» f iv

Fig.

3. -Pictorial

representation

of the energy

landscape

in the

phase

_space for the o.p.s.,

a = 0.5 _case. For P _{~ oJ} the direction C, determined

by

the _sum of all stored pattems, ^is a local

minimum of the Hamiltonian, with E _~ 0. In the P

~ oJ the center of the bowl and all the stored

pattems merge in _a

unique

wide minimum.

60 JOURNAL DE PHYSIQUE I N° I

Figure

4 shows the

plot

of

f (mo)

_{versus mo for} different activities

(a

= 0.2 and

0.4)

for the

p.a.s. case. Here too, as in the o.p.s.

configuration,

the _curves with _same number P of stored pattems superpose and

again

the effect of

increasing

the size N of the _net is _to pronounce the rate at which

f(mo)

increases from 0 to 1. It also

implies

that the size of the basins of attraction decreases with _a. The

explanation

is similar _to the o.p.s., a = 0.5

configuration

there is _a

high

_energy

region

in the

phase

_space,

comprising

now the _a

= 0.5 states, where there _{are not} minima of energy. This

region separates

two multi-hole-bowl-like

regions,

each of them

containing

all the pattems or all the

antipattems and,

_as the

activity

_a

decreases,

the bowls grow

deeper,

_more

energy-decreasing paths

lead from _an initial state to the _center of the bowl and the initial

configuration

should be _nearer to a

given

_pattem in order _to retrieve it _{: more} information is

required

to

distinguish

between correlated

pattems.

Nevertheless it

stiff

represents

an enhancement in

comparison

to the

original

_or modified

Hopfield

model

[4].

I-o

~~~W

n

~ ~

fi

(

,,'

~

'i ~~

0.8

/

^{I ~,'}

~/

l / ?

/ i

--°.6

/

~~P=13

/ 128

cJ

,

j~ _, ^~ /

~

-DA _j ^~ / _/

~~'

/ /

~

d

o-Z / _/ /

Id

/ ~

t' ~/

0.0

DO 0 2 0 4 0 6 0 fi 0

mo a)

1-o

0.fi

--°.6 P=13 128

I i$°

C ^~

0.2

0.0

0 0 0.? 0 4 0 6 o-B 0

mu b)

Fig. ^4. f

(mo

versus m~ for the p.a.s. case with activities _: a) _{a =} 0.4 and b) _a

= 0.2 for _two different values of P and for N

=

128 (O), 256 (6) and 512 (o).

In

figure 5,

we present ^{the results for the} o.p.s.

configuration

for low

activity _pattems

and the

only

effect when

comparing

to the _a

= 0.5 case is the reduction of the size of the basins of attraction.

1.0

II II

o-a

-°.6 P=13 128 ^'

£

c~

-0.4

~

/ /

o-z /

/ ^/

/

0.0

0.6 0.7 o-B 0.9 1.0

mo a)

0

o-a

j°.6

P=13 128

~o.4,

o-z

o-o

0.6 0.7 o-B 0.9 1-o

mo b)

Fig.

5.

f(m~)

_{versus mo} for the o.p.s. case with activities _: a) _a

=

0.4 and b) a = 0.2 for two different values of P and for N

= 128 (O), 256 (6) and 512 (o). Note the different scale _on the horizontal axis.

In

conclusion,

_we

presented

here the results of numerical simulations for the Hamiltonian

equation (5), considering

two

limiting

_{cases :} when both

pattems

^and

antipattems

_are stored

(p.a.s.)

and when

only

the

pattems

are considered

(o.p.s.).

In the _case of _a

=

0.5,

the

capacity

of the net is

greatly improved, specially

^for the p.a.s. nets,

compared

to the

Hopfield

model. Also the retrieval of memories _are

always perfect, I.e.,

the final

overlap

^of a retrieved

state is

always I, reflecting

the fact that the

pattems

are

always

minima of the energy function.

The Hamiltonian function _can be

expanded

in _{a sum} of different orders of interactions

and,

in the p.a.s. case, it _can be truncated _at the second order term,

recovering

the

Hopfield

^model.

As this cut-off is reasonable

only

when the load parameter ^{is low and the}

pattems

are

uncorrelated 11

2, 16],

the critical value a~ = 0.14 may be

regarded

_{as a}

limiting

value for the

validity

of the

approximation implied by

the cut-off at the second order term.

JOURNAL DE PHYMQLLJ -' 2,A J"'Ilk1'>.>2 t

62 JOURNAL DE PHYSIQUE I N° I

A

novelty

^is that, ⁱⁿ the o.p.s.

configuration,

the size of the basins of attraction scales with the number P of stored

pattems,

^because of the bowl-like

shape

of the energy in

phase

_space.

The p.a.s.

configuration

_seems to occupy more

efficiently

the

phase

_space. On the other

hand,

in the o.p.s. case the basins of attraction may become _too small. This reduction in the size of the basins is also verified for low

activity

_pattems, for both p.a.s. and o.p.s. nets.

A limit value for the load parameter a for _a

=

0.5,

_p.a.s. nets has _not been found.

Anyway,

the pattems are

always

zero-energy states of _a

non-negative

Hamiltonian and hence the

definition of _a limit value for _a may

require

_some

adjustment

^{in this model} : for

example,

_a

limitation may be

imposed by

the

velocity (or

_some other

parameter)

in

retrieving

the

pattems ^{rather than}

by

the

ability

in

addressing them, although

_up to now we did _not detect such effect. Work in this direction is

presented

elsewhere

ii?].

References

[Ii ^{MCCULLOCH W. S. and PITTS W.} A., Bull. Math. Biophys. 5 (1943) lls.

[2] HOPFIELD J. J., Proc. Natl. Acad. Sci. USA 79 (1982) 2554.

[3] HEBB D. O., The

Organisation of

Behaviour (New York

Wiley,

1949).

[4] AMIT D. J., GUTFREUND H. and SOMPOLINSKY H.,

Phys.

Rev. A 32 (1985) 1007 Ann.

Phys.

NY 173 (1987) 30

Phys.

Rev. A 35

(1987)

2293.

[5] GARDNER E., J. Phys. A 20 (1987) 3453.

[6] KANTER I. and SOMPOLINSKY H.,

Phys.

Rev. A 35 (1987) 380.

[7] HORN D. and USHER M., J.

Phys.

France 49 (1988) 389.

j8] ABBOTT L. F. and ARIAN Y., Phys. Rev. A 36 (1987) 5091.

[9] NADAL J. and TOULOUSE G., Nenvork 1 (1990) 61.

[10] PEREz-VICENTE C. J. and AMIT D. J., J.

Phys.

A 22 (1989) 559.

[I Ii PEREz-VICENTE C. J., Europhys. Lett. lo (1989) 627.

[12] DE ALMEIDA R. M. C. and IGLESIAS J. R., Phys. Lett. A146 (1990) 239.

[13] FORREST B. M., J. Phys. A 21(1988) 245.

[14] PENNA T. J. P. and DE OLIVEIRA P. M. C., J.

Phys.

A 22 (1989) L719.

[15] DE OLIVEIRA P. M. C.,

Computing

Boolean Statistical Models (World Scientific

Publishing,

1990).

[16] ARENzON J. J., DE ALMEIDA R. M. C. and IGLESIAS J. R.,

preprint (1991).

[17] PENNA T. J. P., DE OLIVEIRA P. M. C., ARENzON J. J., DE ALMEIDA R. M. C. and IGLESIAS J. R., Int. J. Mod.

Phys.

C 2 (1991) 7 II.