Short-term memory in a sparse clock neural network

HAL Id: jpa-00246756

https://hal.archives-ouvertes.fr/jpa-00246756

Submitted on 1 Jan 1993

HAL is a multi-disciplinary open access archive for the deposit and dissemination of sci- entific research documents, whether they are pub- lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Short-term memory in a sparse clock neural network

S. Semenov, A. Plakhov

To cite this version:

S. Semenov, A. Plakhov. Short-term memory in a sparse clock neural network. Journal de Physique

I, EDP Sciences, 1993, 3 (3), pp.767-776. �10.1051/jp1:1993161�. �jpa-00246756�

J.

Phys.

I France 3

(1993)

767-776 _MARCH 1993, PAGE 767

Classification

Physics

Abstracts

05.90 64.60 87.10

Short-term memory in

_a

sparse clock neural network

S. A. Semenov and A. Yu. Plakhov

Institute of

Physics

and

Technology,

Prechistenka Str, 13/7, Moscow119034, Russia

(Received 7

September

1992,

accepted

21 October 1992)

Abstract. We propose a model of short-term

(working)

memory in _a clock neural network. The retrieval dynamics of _a

strongly

diluted version of the model is solved. The related

phase diagrams reflecting

fixed

point

retrieval behaviour of the system are also obtained and discussed.

1. Introduction,

A direct and

intuitively

evident way to avoid

overloading

in

Hopfield-type

neural networks is that of _an introduction of

non-additivity

into

leaming

rules which saturates neuronal

couplings

and, in

fact,

is

responsible,

for

palimpsestic

behaviour of associative memory models

[1-3].

In this

regard

several

leaming

schemes for

binary

networks _were

proposed [2, 4, 5], analysed [3- 6]

and discussed from _a

biological viewpoint [7]. Newly developed

models have _even closer

relations with

psychological phenomena ^[8].

It is of interest therefore _to

investigate

manifestation of short-term memory effects in other

important

attractor-like neural network

models,

such _as, for

instance,

clock _or Potts models. In recent years, a clock model which

exploits

a Hebb-like

learning prescription

has been treated

[9, 10].

As it _was

found,

the

system,

^when

being

overloaded

by

learnt

pattems,

^{falls to the}

state of total memory ^{confusion. It thus} seems valuable _to invent _a tractable

learning procedure

that

prevents

memory

overloading

in _a clock neural network.

In this paper we

present

^and

analyse

_a model of _a clock neural network

exhibiting properties

of short-term

(working)

_memory. ^In our model intemeuronal

couplings

_are

given by complex

numbers with fixed

modulus,

and the

leaming

of any ^new pattem ^{consists of Hebb-like}

modification and then renormalization of each

coupling.

This

factually

leads _to memorization of new pattems while

simultaneously forgetting

the most ancient _ones and _{as a} result the total

length

of stored pattems ^{list remains} constant. We conclude in effect that main features inherent to the models of

working

_memory

designed

for

binary

neural networks such _as, for

example,

an existence of critical and

optimal embedding strengths

_are

preserved

in _our model.

The paper is

organized

as follows. In the _next section _we describe _a

general

model which will be solved in the third section in the limit of extreme dilution. In the forth section

stability analysis

is carried out

providing phase diagrams

in the

parameter

space of the model.

Then,

in section

5,

_we

analyse

ⁱⁿ detail the deterministic

dynamics

limit. The _paper finishes with

concluding

remarks.

2. Definition of the model.

Let _us consider _a network of N

planar spins (neurons).

The _state of each _neuron is

given by complex

variable s~,

[sk(

_"

I, (k

=

I,

,

N

).

The _state of the system ^{is then described}

by

vector s =

(si,

,

s~). Spin dynamics

is

govemed by

^{local field} of the form

N

~k(S)

"

I

_{l~ki St}

>

f=1

with

complex couplings T~i.

We

imply

that stochastic evolution of the system ^is

given by

^either

parallel

or

sequential

heat bath

dynamics

with time

step

At

by assuming

transition of _neuron k _to the _{state z} in the next time moment t + At _to be

given by probability density

~

(s~(t

₊ At

)

= z = p

(z v~(t ) ),

where

exp(ji

Re

(z*

_V»

&(jzj I>

,

~~~~~~ 11)

~~~~p

R~

~w*v»&(jwj I)dw

~

~

v~(t)

=

z T~i si(t> (2>

Here asterisk marks

complex conjugation

^and the

couplings

_T~t

=

K~t J~t

combine the effects of

leaming (J~t)

and dilution

(K~t).

The factors

K~t

are chosen _to be

independent

random

variables with distribution

p

(Kki)

=

(C

IN

(K~t

I/

$)

+

(I

C/N

(K~t),

here C is the _mean number of

couplings

of each _neuron. Since

K~t

and

Kt~

^do not correlate with each other the dilution is

asymmetric.

The parameter fl ⁱⁿ

(I)

is the inverse temperature

determining

the level of stochastic noise. For

parallel (synchronous) updating

of all the system

a time increment At

=

I is

implied. Sequential dynamics

consists in random choice of network's _neuron and

updating

it in time interval At

=

N

according

to stochastic

dynamics (1)-(2).

In the deterministic

limit, fl

= oJ, the

dynamics

becomes

simply s~(t

₊

At)

=

v~

(t )/

_v

~

(t )

During

^the

learning

_process random

pattems

^which are nominated

configurations

N

s~,

_v

= ,

1, 0,

1,

, p chosen

independently

with uniform distribution

©(s)

₌

fl g(s~),

k I

g(s )

=

(2

_ar

)~ ( Is

I

),

_are

sequentially presented

to the network. If _one tries _{to store} the

pattem's

sequence

using

Hebb-like

prescription

~ki

"

iS~ ^Si'

v

the memory deteriorates _{at some} critical

loading

and the system ^{loses the}

ability

to retrieve any pattem

taught [9, 10].

To _overcome the _state of memory

overloading,

it _seems to be reasonable to restrict the range of

possible

values of

complex couplings

in _a

quite

natural way

by assuming J~t

to take the values _on the unit

circle,

I,e.

J~i

=

I. Below _we propose an

embedding

scheme

enabling

_us to memorize the most recent pattems ^{in the} sequence leamt. This is achieved in

step-by-step «embedding

and renormalization» of

couplings.

^That

is,

at the time of

acquisition

^{of the v~th} pattem, the

couplings

_{are to} be

changed by

~ ~

J(I

+

es( s)

Jki

"

~ ~

~

(3)

(J~t

+ esk si

N° 3 SHORT-TERM MEMORY 769

where

e is the

embedding strength.

In _{contrast to} Hebb-like additive

leaming proposed by

Noest

[9],

in which both

amplitudes

and

phases

of

couplings J~t

are

modified,

the rule

(3) implies

that the information is stored in the

phases fl~t

₌ arg

J~t only.

Note that the

learning procedure (3)

and the network

dynamics (1)-(2)

_are invariant with respect to

global

rotation of the

system,

_{s~ - cs~,}

[c[

= I.

Following

Noest

[9]

let us define _{an «}

overlap

_» between network

configurations

_s and

S'by

m(S, S'>=

^'

~l Z Sksl*

For random noncorrelated

configurations,

we have _m N ^~~~ while _ones, which coincide with each other after _some

global rotation, specify [m

=

1.

3.

Extremely

diluted network.

If the _mean number of unbroken bonds of _neuron,

C,

becomes of order

logN

_as

N - oJ, one may

neglect

correlations in the _{« ancestor} tree » and _an

analytical

treatment of

dynamics

_can be used

(Derrida

et al.

[I II).

In the

following

_we solve the

dynamics

of such _an

extremely

diluted network

(1)-(3)

^with

embedding strength

_e in

teaming

rule

(3) going

to zero.

We will be interested in storage ^{and retrieval of the} ~p + I

)-th pattem

from the end of pattem sequence..

,

s~ ~,

s°,

s~,

.,

sP,

I-e- the pattem

s°.

First of all, _we will fix indices k and

f

_and

concentrate on the contribution of the pattem

s°

_{to the formation of the}

coupling J~i.

It is easy to show that in view of the

independence

^and the uniform distribution of

(s( s)*,

v =

,

1, 0, 1,

,

p)

on the unit circle every

given

increment

4(1m

fl

(t+ flit

does _not correlate with the

previous

ones.

(We

remind that

fib

= arg

J(I).

Indeed, the

quantity

J(/ _s( s)*

is

uniformly

distributed on the unit circle and does _not correlate with all

quantities

sf~~(sf~~)*,

pm v and hence with

Jft.

From

this, according

to

(3),

it follows that

J(t+ J(/

=

(I

+

eJ(/ _{s( s)}

^* _{I +}

eJ(/ _{s( s)*}

_{does not correlate} with all

Jfi,

_{p w v.} In terms of arguments ^it means the above stated. From this fact _one

immediately

_gets mutual

independence

of all the increments

(4j,

_p

=. _,

1, 0,

1,

,

p).

As _e

-

0, by taking

linear

part

^of

expression (3)

we

get J(t+ J(/

= I ₊ I _e Im

(J(/ s( s)* )

+ O

(e~), (4)

or, in _terms of arguments,

4(1

= E Im

(J(/ _{S( S)~)}

₊ O

(E~).

Variance of

4(I

is therefore estimated _as

e~/2

+ O

(e~).

The

resulting change flit+ fill

which is obtained after

storing

_{of p} most recent pattems

s~,.

,

sP is thus

given by

the _sum of _p

independent

random

quantities flit+~ fill

p

=

I Al

with total variance

pe~/2

₊

O~pe~).

a i

Consequently,

for

large

_p the

quantity f~t

₌

~pe~/2)~~/~ (flit+~-flit)

is

approximately

Gaussian with _{a zero mean} and unit variance.

Introducing

the notation _y

m

p/C

for the short-term storage

capacity

and _e;=

eC~'~

_{for the}

reduced

embedding strength (further

_{on we} Will call _e

embedding strength

to be _more

concise),

we now

impose

the constraint that both _y and _{e are}

kept

fixed in the limit _p, C

- oJ. This

implies

for _e

scaling e~C~~~~

_In these terms _one gets

flit+~- fl(t

=

(ye~/2)~~~ (~t.

Retuming

to

complex

notations and

using

formula

(4),

_{we can} write down

~~

"

~(i

_~XP

(' ~

~ki )

_~

"

J~i

+

fi

_[S~

Si~ (J~i)~

_S~~

Sil

₊

(E~)~

X

eXp(I /~ _(ki)

2 C

We suppose ^next that the initial network's state

s(0)

has _a

macroscopic overlap only

with the pattem

s°. Studying

the retrieval process of pattem

s°,

_we

_represent

the local field in the

moment t, in the form

vk(t>

=

iBk(t>

+

fk(t>i si

>

(5>

where

Bk(t

₌

fi I Kki Si~

_S(

(t )

_eXp

(I /~

~ki )

2 C >

~

j

and

~k(t)

=

§ _I

(4i

~

~ (4i)~

_S~~

_Si

₊ °

(~ )j ^~XP1' ~

~ki )

St

(I)

>

(~)

~fj

where the _sum

z

is

given

_over all

i's

connected _{to neuron}

k, K~t

# 0.

B~(t)

in

(5)

_can be

~fj

interpreted

as a

signal

from pattem

s°.

_{It is}

represented by

_a

self-averaging quantity

^with the

mean

equal

to

Bm(t),

here

m(t)

is _a

time-dependent overlap

between

configurations s(t)

and

~o

m(t)

#

( Sk(t)

_S~~

~

k=1

and

B

=

(e/2)(exp(I fi ())

=

(e/2) exp(- ye~/4)

<

( .)

,

signs

_{average over} normalized Gaussian variable

D. f~(t)

ⁱⁿ

(5) plays

the role of noise which, as can be shown

using

standard arguments

[11],

is

represented by

the _sum of

uncorrelated items that

approaches,

_as C

~

log

N

- oJ, a

complex spherically

distributed Gaussian

quantity

with _{zero mean} and unit variance, I,e.

f~(t)j~)

₌ ^I.

Let _us consider the _case of

parallel dynamics

with At

=

I.

By using

the

expression

for conditional transition

probability (I),

the thermal average of

s~(t

+

I)s(*

under fixed

f~(t ) equals c s(*

_zp

(z s((Bm _{(t )}

₊

_{f~(t )))}

dz. Due to rotational invariance of the function _p, p

(z

v

)

_{m p}

(sz

sv

) provided

_s

=

I,

after substitution _w

=

s(*

z we can rewrite the

expression

for thermal average as

s~

(t

₊ I

s(*

= wp

(w

Bm

(t

+

f~

(t

)

dw

c

Taking

_average over

f~(t),

we

get immediately

(s~(t

+ I

s(*)~

^{= F}

^{(m(t)) (7)}

N° 3 SHORT-TERM MEMORY 771

with

F(m>= lwjp(wjBm

⁺

^f»~dw,

c

where

(. )

~ ^means

averaging

_over normalized

complex spherically

distributed Gaussian

variable

f.

Without any loss of

generality

_{we may assume} the initial

overlap

_m

(0 )

to be real and

positive (it

_can

always

be achieved

by

_some rotation of the whole

system). Obviously,

the function F

(m )

takes real

positive

values for

positive

_m and hence the

overlap dynamics

_occur

on

positive

half-axis.

As _a

result, taking

into account that _an

overlap

is _a

self-averaging quantity,

in virtue of

(7),

it

straightforwardly

follows _an evolution

equation

in terms of

overlap

m(t

+ I

=

F

(m (t (8>

In the _case of

sequential dynamics

similar consideration

gives

evolution of the form

=

F

(m(t m(t> (9>

Evidently,

iterative

(8)

and differential

(9) equations

have the same fixed

point

solution m*

= F

(m*)

that defines retrieval

overlap.

In

principle,

this solution _can be obtained

by

numerical

integration

which appears ^to be time

consuming

for _nonzero temperatures. ^We

therefore confine ourselves _to the _zero temperature ^limit

only

which is

relegated

to section 5.

For

arbitrary temperatures,

^the

stability

of trivial solution _m

= 0 will be examined

(Sect. 4).

4.

Stability analysis

and

phase diagrams.

Since F

(m)

is _a

strictly

_convex function

(that

_can be checked

through

^routine

calculation)

the condition for the existence of _a

unique

nontrivial solution _m * of evolution

dynamics (8)

_or

(9)

is then

F'(m

=

0)

_> 1.

Thus,

the critical relation

(here

_x

= 3l

(f )

is

denoted)

defines _a critical surface in the parameter space

(e,

_y,

fl)

at which the second order transition to memory deterioration takes

place.

To obtain this relation in _an

explicit form,

_we first introduce the notation

f

= x +

iy,

z =

exp(14 )

in which the

expression

for aplax in

(lo)

takes the form

~~

= exp

[fl (x

cos

4

_{+ y sin ~fi}

)]

_exp

[fl (x

_cos it + y sin it

)] fl (cos

4 _cos

it ) dir

x aX

~

1@

-2

x

-«

exp

~p (x

_cos

it

+ y sin it

)] dir (I1)

Then

using polar

coordinates _{x +}

iy

=

rexp(10)

and

talcing

into account that

(. )~

=

m «

(2 ar)~ dr~ exp(- r~)

do..

,

after substitution

(11)

into

(lo)

_one obtains

-

«

? ^°~ _{dr~ exp(- r~> j"}

_do

_{j" d#R}

(r,

o,

#

_CDs

#

= i

,

o _«

«

where

R(r,

0,

4 1«

= exp

[fir

_cos

(4

0

)]

_exp

[fir

_cos (11 0

)] (cos 4

_cos it

)dir

_x

«

1«

^-2

x

- exp~pr

cos (11 0

)] dir

«

Replacing 4

0

by

_x and

performing integration

_over

4,

_{X we}

finally get

~~

e~ Y~~~

l~ ^{dr~ ~~~(1 ~~~~~~ j}

= ,

(12)

4

~

I~(fir)

here

I~ (x ) 1«

= ar exp

(x

_cos

4

cos

(n4 d4

is the modified Bessel function of n-th order.

o

The solutions to

equation (12)

can be obtained

numerically providing

the critical surface in the parameter _space. In

figures

^1-3 we present

phase diagrams corresponding

to

sectioning

this

surface

by planes

with _one of the parameters

being

^fixed.

The critical lines

y~(e

_; fl

=

Cst)

_are

plotted

in

figure

I for several values of the noise

parameter fl.

The function

y~(e

_;

fl

₌

Cst)

reaches its maximum at the

point

_e~~~(fl

), shifting

towards

larger

values of _{e as} temperature increases, and becomes _zero at some

point

e~(fl (critical embedding

st

j~th

^at

^given temperature). e~(fl

increases with

temperature

from _e~

=

e~(fl

=

oJ)

=

4/ _ar,

behaving

_as

fl~~

_as

_fl

-0

(it

can be

straightforwardly

deduced from

(12)).

In the limit

e - oJ, at any temperature, ^{the maximal} storage

capacity

vanishes _as

y~(e

fl ₌ Cst e~ ^~ ln _e. In

fact,

the storage

capacity

reduces

sufficiently

with temperature increase

(see Fig. I) indicating

noticeable

sensitivity

of memory

functioning

to

dynamical

noise.

O.1 6

~O.12

- U O Cu O

° O.08

© Oh

$

O

$

O.04 ^'

ooo

ini>erse

_em

strength

Fig.

I.- Critical lines in

(e~~- _y)-plane

_at fixed noise parameter p (from top to bottom p ₌ w, 10, 4, 2, 1, 0.5). Nonzero retrieval regions are below the corresponding curves.

N° 3 SHORT-TERM MEMORY 773

A

family

of critical _curves in the

(e fl )-plane

is

plotted

for several fixed

yin figure

2.

For the choice _y

=

0,

the critical

temperature

goes ^to

infinity

as e, e - oJ. It

implies

that _at

arbitrarily high temperatures

an intensive number of

pattems

p =

o(C)

can be stored for

sufficiently high

values of _e.

Otherwise,

if _{y >} 0 is

assumed,

there exists _a critical

temperature

above which the

pattems'

list of

length yC

cannot be retrieved for _{any e.} As _y increases the

area of _nonzero retrieval

drops

and vanishes at y = yo = 0.145.

4.OO

3.OO

~

©

~$ (

2.OO

Cu

E

©

i .oo

o.oo

invhrse embedding strength

Fig.

2. Critical lines in (e~ p

~)-plane plotted

for several values of storage

capacity

_y (from _top to

bottom _y

= 0, 0.01, 0.05, 0.I). Nonzero retrieval

regions

_are below the

corresponding

curves.

Critical lines below which _nonzero retrieval appears are drawn for certain values of _e in the

(y fl~~ _{)-plane (Fig. 3).}

When

embedding strength

exceeds e~, a finite _area of _nonzero

retrieval arises _near the

origin. Figure

3

displays

_a characteristic

change

of the

shape

of retrieval

regions

with _an increase of _e. As _e

- oJ, the

slope

of the critical _curve grows

infinitely.

5. Zero

temperature

^limit.

In the _most

important

case of absence of

dynamical noise, fl

= oJ, the distribution

density

_p

becomes a delta function

p(z[v)=&

_z- ^~

(V(

and F

(m)

takes the form

F

(m

=

~"~ _~

~

(Bm+f(

_~

(13)

JOURNAL DE PHYSIOUE I -T 3, N'3 MARCH 1993

1.60

1.20

~

©

li

~j ^O.80

E

CL

©

O.40

o.oo

storage capacity

Fig.

3.- Critical lines in (y

fl~~hplane _pictured

_for

a set of _e values (from _top to bottom

e =

6.0, ED ^" 3.7, 2.5). Nonzero retrieval

regions

_are below the

corresponding

_curves.

Using polar

coordinates

(r,

4

),

Bm ₊

f

₌ r exp

(14

_{we can} rewrite

(13)

_as

lm

^«

F

(m

=

(2

_ar

)~ dr~ _d4

cos

4

_exp

(- r~

+ 2 Bmr _cos

4

B^~

m~ )

,

0

-

«

and after

integrating

_{over r we get} the

expression

for F

(m)

«

F

(m)

=

(2

_ar )~

exp(- B~ m~) d4 (1

₊

ar~'~Q exp(Q~) _II

₊

W(Q)I)

cos

4

,

(14)

-

«

lx

where

Q

= Bm cos 4 and W

(x )

= ar

~/~ exp

(- t~ )

dt.

x

The critical relation

(12)

is

greatly simplified

at zero temperature

giving

_a critical line of the form

y~(e)

= 2

e~~In (are~/16).

The system ^{is thus} able to store p =

y~(e )C

>0 last pattems ^{if the}

embedding strength

exceeds threshold value e~ =

4/

7~=

2.257. The function

y~(e)

reaches its maximum

yo =

y~(eo)

=

ar/8 _e =0,145

(optimal

short-term storage

capacity)

_at

optimal embedding

strength

_{eo =} 4

/~

= 3.721 and y~ vanishes ~p~ =

o(C ))

_{as e}

- oJ

(see Fig, I). Evidently,

a

qualitative picture

of memory

performance

bears _a

strong

resemblance _to diluted versions of the

Hopfield-Parisi

model and similar _ones

[5].

It is

interesting

to note that the ratio between

doubled

optimal

short-term

storage capacity

and critical

storage capacity

of _a sparse

phasor

model

by

Noest

[9] (the coupling Jki

defined

by (3)

has

only

_one

degree

of freedom

fl~t

instead of _two in the

phasor model)

is

exactly

_{e~ ~,}

coinciding

with _an

analogous

ratio for

strongly

diluted version of

binary

models

[5].

N° 3 SHORT-TERM MEMORY ₇₇₅

Fixed

point

solutions m*

=

F

(m*)

of retrieval

dynamics

at zero

temperature

^have been obtained

numerically

for varied choices of storage

capacity

_y and

embedding strength

_e.

Retrieval

overlap

m* _{as a} function of _y is

displayed

in

figure

4. The

system

^exhibits a

continuous transition to zero retrieval

phase

when the short-term

storage capacity

reaches its

critical value

y~(e)

=

y~(e fl

= oJ

).

The

optimal overlap

value is

m?

=

m*(y

=

0; eo)=

0.889. If _e grows ^from e~ ^to eo, the maximal

storage capacity y~(e; fl

=

oJ)

enhances

(see

also

Fig. I). Otherwise,

for

high

intensities of

learning,

e > eo, it reduces

but, simultaneously,

the retrieval

quality improves greatly

at low

storage capacities

_so that

m*(y e)

- I when _e

- oJ, y =

O(e~~). Thus,

_an increase of

embedding strength

leads to

improvement

of

recalling precision

of the _new

pattems

at the expense of

maximal storage

capacity.

1.oo

*i O.80

CL O

j

O.60

>

O

)

_O.40

©

°C

$

~ 0.20

o.oo

m 6

storage capacity

Fig.

4. Fixed

point

solution m* of retrieval

dynamics

(8, 14) _{versus storage}

capacity

_y for certain values of the

embedding strength

_e at fl

= w (from _top to bottom _e

= 6.0, _so

= 3.7, 2.5).

Retrieval

overlap

_versus inverse

embedding strength

^{is drawn for selected values} of storage

capacity yin figure

5. For _y

>

0,

the retrieval

overlap

becomes _{zero at}

sufficiently large

_e

because the

progressive

increase in

embedding strength produces

_{a more} intensive

forgetting

of the _most ancient pattems so as that the _set of p =

yC

_{pattems cannot} be retrieved _{as a} whole.

6.

Concluding

remarks.

We have

presented

the model of

working

_memory in _a clock neural network which has been solved in the limit of extreme dilution. Our

analysis

shows that the system possesses nonzero

short-term storage

capacity,

when

embedding strength

exceeds _some threshold

value,

and demonstrates continuous transition to memory ^{loss at} critical surface in the parameter _space. It

oo

~.

o

O.80

CL O

[

O.60

>

O

)

_O.40

© 'C

$

~ O.20

O.OO

inverse

_e

stre

Fig.

5. Retrieval

overlap

m* _{as a} function of inverse

embedding strength

e~ for several values of storage

capacity

_y at fl

= w (from _top to bottom _y

= 0, 0.01, 0.05, 0,1).

tums out that the

qualitative picture

of memory

performance

^is

essentially

the _{same as} it _was found for

binary

nets

[5].

It is worth

mentioning

that

proposed learning algorithm

_seems to

acquire

_necessary

simplicity

to be

implemented technologically

on the base of coherent

optics technique [12].

In the

end,

it should be noted that neural network models

using

n-vector

spins

as neurons can

be

developed

and treated in close

analogy

with the present consideration.

References

[I] HOPFIELD J. J,, Proc. Nail. Acad. Sci. USA 79 (1982) 2554.

[2] PARISI G., J.

Phys.

A 19 (1986) L617.

[3] HEMMEN VAN J. L., KELLER G. and KUHN R., Europhys. Lett. 5 (1988) 663.

[4] NADAL J. P., TOULOUSE G., MEzARD M., CHANGEUX J. P. and DEHAENE S.,

Europhys.

Lett. 1 (1986) 535.

[5] DERRIDA B. and NADAL J. P., J. Stat.

Phys.

49

(1987)

993.

[6] MEzARD M., NADAL J. P. and TouLousE G., J. Phys. France 47 (1986) 1457.

[7] NADAL J. P., TOULOUSE G., MEzARD M., CHANGEUX J. P. and DEHAENE S., in Computer

Simulation in Brain Science

(Cambridge Cambridge University

Press, 1988) _p. 221.

[8] WONG K. Y. M., KAHN P. E. and SHERRINGTON D., J.

Phys.

A 24 (1991) ll19.

[9] NOEST A. J., Europhys. Lett. 6 (1988) 469.

[lo] COOK J. J., J. Phys. A 22 (1989) 2057.

[ll] DERRIDA B., GARDNER E. and ZIPPELIUS A., Europhys. Lett. 4 (1987) 167.

j12] ANDERSON D. Z., in Proceedings of the Conference _on Neural network models for

optical

computing,

13-14 January 1988, Los Angeles, Califomia, I. Athale, A. Ravindra Eds., p. 417 and references therein.