Information processing in synchronous neural networks

HAL Id: jpa-00210669

https://hal.archives-ouvertes.fr/jpa-00210669

Submitted on 1 Jan 1988

HAL is a multi-disciplinary open access archive for the deposit and dissemination of sci- entific research documents, whether they are pub- lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Information processing in synchronous neural networks

J.F. Fontanari, R. Köberle

To cite this version:

J.F. Fontanari, R. Köberle. Information processing in synchronous neural networks. Journal de

Physique, 1988, 49 (1), pp.13-23. �10.1051/jphys:0198800490101300�. �jpa-00210669�

Information processing ⁱⁿ synchronous ^{neural networks}

J. F. Fontanari and R. Köberle

Instituto de Fisica e Química de São Carlos, Universidade de São Paulo, Caixa Postal 369, ^{13560 São} Carlos, SP, ^Brasil

(Requ ^{le 16} juin ^1987, accepté ^{le 16} septembre 1987)

Résumé.

Nous obtenons le diagramme ^de phase du modèle de Little quand ^le nombre p d’échantillons mémorisés croit comme 03C1

03B1N, ^{où N est le} nombre de neurones. Nous dédoublons l’espace ^de phase ^de façon ^à accommoder des cycles ^de longueur deux dans le cadre de la mécanique statistique. Utilisant la méthode des répliques, ^nous déterminons le diagramme ^de phase ^{incluant un} paramètre J0 pour contrôler

l’apparition ^des cycles. ^La transition de phase ^{entre les} phases para- ^et ferromagnétiques passe du second ordre

au premier ^{ordre au} point tricritique. ^La région ^de recouvrement de l’information est un peu plus grande que dans le modèle de Hopfield. ^{Nous trouvons} également ^une phase paramagnétique ^{à basse} température qui ^a

des propriétés physiquement inacceptables.

Abstract.

The phase diagram of Little’s model is determined when the number of stored patterns p grows ^as 03C1

03B1N, ^{where N} is the number of neurons. We duplicate phase space in order ^to accomodate cycles ^of length

two within the framework of equilibrium statistical mechanics. Using ^the replica symmetry ^{scheme we} determine the phase diagram including ^a parameter J0 ^{able to} control the occurrence of cycles. ^{The second}

order transition between the paramagnetic ^and ferromagnetic phase becomes first order at a tricritical point.

The retrieval region ^{is some what} larger ^{than in} Hopfield’s ^{model. We also find a low} temperature paramagnetic phase ^with unphysical properties.

Classification

Physics ^Abstracts

87.30G

64.60C

75.10H

89.70

1. Introduction.

Recently ^methods developed ^{in the} study ^of equilib-

rium statistical mechanics of spin glasses [1, ^{2, 3,} 4],

have been applied ^to investigate information proces-

sing and retrieval in neural networks. Although

there is a long way ^to go, if reasonably ^realistic biological systems ^{are to be} described, the models

we are able to control, do exhibit a number of

interesting ^{features, which} makes their study ^a

worthwhile endeavour. Properties ^{such as} fault toler-

ance to errors, information storage and retrieval due

to implementation ^of auto-associative memories,

etc. have been shown to arise as a consequence of the existence of an infinite number of ground ^states

in ^a spin glass interacting ^via long ranged ^forces.

In this paper ^we study Little’s model [5 ^{based on} synchronous update ^{in the limit, when an infinite}

number of patterns ^{is to be stored} [6]. ^In contrast to

assynchronous dynamics (such ^as Monte Carlo up- date in Hopfield’s model), ^{when all states} change simultaneously ⁱⁿ parallel, ^the system may exhibit

cycles. ^It is indeed easy ^to show that Little’s model

with symmetric couplings ^{has limit} cycles ^oft length

two in addition to fixed points.

The question arises then as to how can equilibrium

statistical mechanics describe ^a system ^with cycles [2]. ^{As we will see, since in our case the} cycle’s length ^{is two, we have to} duplicate ^the phase space

[7]. The concomitant proliferation of order par- ameters allows the adequate description ^{of this more} complicated ^behaviour.

In section 2 we calculate the free energy and derive the equations for the order parameters, ^which

are discussed and solved for special ^{cases in sec-}

tion 3. A discussion is presented ^{in section 4.}

2. Free energy and order parameters.

Little’s model consists of a network of N neurons

described by spins S, = ^{± 1, i}

^1,

^N. The transi- tion probability W (J/I ) ^{from state I}

{5,} ^{at time}

t to state J at time t + 1 is given by

Article published online by EDP Sciences and available at http://dx.doi.org/10.1051/jphys:0198800490101300

where

For symmetric couplings Jij

Jji, this leads to a

stationary distribution of states given by e- H, ^with

Here f3 ^{is a measure of the} noise level in the system ^{and the} couplings ^are

The p patterns (prototypes) {ç r ; i ^{= 1, ...,} N }

J.L =1, ..., p ^are quenched, independent ^random

variables taking the values ± 1. Due to the coupling Jij ^{these patterns are fixed points of the dynamics at}

T=0, poo.

In order to evaluate the trace Tr e- OR _{we use} the W

identity

where aj = ^± 1, j

1, ^{..., N are a set of} duplicate Ising ^variables.

Following Amit et al. [3] ^{we now} compute ^the quenched free energy under the assumption ^{that a}

finite number s of the overlaps ^{n "}

remains finite as N --+ oo. The

quenched free energy per spin ^is given by ^the replica

method :

where

Introducing ^the replicated ^variables sf, (T f, ^p

^1,

..., n ^we write (Z") ^as

where

with Z°

Z v for cp * s and Z°

Z w for _cp > s.

Let us first evaluate Z’. Performing the average ^over the g¡, ^we obtain the factor

If we expand ^the log cosh, keeping ^{terms of order} tiN ^{we get}

Inserting ^this expression into Z’ and performing ^the integral ^over m:, n: and f p 0 ^{we obtain finally}

where Q, ^{P and R are} symmetric n ^{x n matrices} given by

where p pu ^and r pa ^{are defined} only ^{for p =1= (T and --AL-} have omitted trivial multiplicative ^{constants in} equation (2.12).

Now we proceed ^to evaluate Z v3. First we rescale m, n and f in order to obtain ^a well defined

thermodynamic ^{limit as}

Thus

Now use self-averaging ^{in the} identity

to replace 1 _{E by and} ^{use the result in} equation (2.15). ^{With this} expression for Z v and equation Ni

(2.12) ^{for Z’ we obtain} finally

where

with the notation m.

In the limit N - oo the integrand is dominated by

its saddle point, furnishing ^the following free energy per spin

with

In this paper ^we mainly discuss the replica symmet- ric theory, ^{in which we use the} following parametri-

zation

With this ansatz the n - 0 limit of the terms in

equation (2.19) may be explicitly performed. ^We get

[8]

where

Finally ^{we have to} compute

Using ^{the ansatz} (2.21), ^we rearrange HI; ^into

Now we linearize the quadratic ^{terms in the} spin

variables by ^Gaussian transformations, ^{to obtain}

after insertion into equation (2.23)

where

and stands for the average ^over the 6,’s ^and

over three Gaussian variables Zi, i

^{1, 2, 3 with}

mean zero and unit variance.

Putting everything together ^we get for the free energy the final expression

Minimizing f with respect ^{to the} parameters f, ^{m, n,} p, p, ^{r, r,} ql, ql, ^qo, 40 yields ^their interpretation ⁱⁿ

terms of the original ^variables Si, (T and );’ together with the order parameter equations. We thus obtain within the replica symmetric theory :

i) ^the macroscopic overlaps ^between equilibrium ^{states and} prototypes :

ii) ^the Edwards-Anderson order parameters

iii) ^{the total mean} square random overlaps ^with p-s patterns

where > ^s.

The 8 + 3 s coupled equations for the order parameters ^{are :}

where

3. Solution of order parameter equations ^and phases.

In general ^{it is} impossible ^{to solve} explicitly ^the coupled equations for the order parameters, ^{but in} special limiting ^{cases this can} be done. In the

following ^we will do this in order to obtain relevant sections of the phase diagram ^and discuss the nature of possible ^solutions.

3.1 PHASE ^AT T = 0.

The sign ^of Q plays ^an important role, ^{if we want to take the limit} J3 --+ ^oo. Accordingly ^we distinguish ^{three cases.}

for equations (2.28) :

where

and

Since _{qo =1} we are at a fixed point ^{of the} dynamics, ^as opposed ^{to a limit} cycle.

In order to evaluate the P -+ ^{oo limit we use the} following ^identities

It is now straightforward ^to get

These are the same equations ^{as obtained} by ^Amit

et al. [3], for the T = 0 limit of Hopfield’s ^model.

This is to be expected, ^{since at a fixed} point equation (2.2) may be rewritten ^at T

0 as

The free energy ^at T

0 becomes

Which differs from Hopfield’s ^model only by ^the

term Jo and Amit et al.’s analysis ^{for this case can be}

taken over with a slight change ^{in notation.}

3.1.2 Q ^0.

^The equations ^{become now :}

where

and

Since qo = - 1, ^{we are at a} complete cycle, ^all spins changing sign ^{at each} update. Evaluating ^{the limits}

in equations (3.6) ^we get

The only ^{solution of} equation (3.7) ^{for m ° is} m’ = 0, yielding

with the energy being given by

Since c is positive semidefinite, ^this spin glass

solution exists only ^for a > 2/ 1T ^and 7() (a /2 ^1T )1/1, ^{the last} inequality resulting ^from

Q0.

If we relax the implicit assumption ^{(3.8) of}

c oo, ^we also encounter a paramagnetic phase ^with

p

0, implying p

^{0, c --+ oo} with energy

which is always less than the spin-glass energy (3.9).

Whether fixed point ^or cycles ^{are realized} depends

now on the value of Jo, selecting the lower of two

energies equation (3.5) ^versus (3.10).

3.1.3 The case Q = 0 gives identical results to

Q > ^0.

^The resulting phase diagram is shown in

figure ^1, where all transitions are first order.

Fig. ^1.

^Phase diagram ^{in the} a - Jo plane. a bounds

the region ^{where retrieval states} appear, which become stable inside the F region. _{a g} separates ^the p and SG

phases. ^At Jo

^{0, we} get a, ^- 0.138 and _{a F} ⁼⁼ 0.051 as in

Hopfield’s ^model.

3.2 PHASES FOR a

0.

The a --+ 0 limit of

equation (2.26) yields

with the order parameter equations

from which ^we easily ^obtain

Since the lefthand side is > 0, ^{whereas the} right ^hand

side is 0 we see that the only solution is

In the following ^{we will} study retrieval solutions of the Mattis type only :

n now satisfies

Expanding ^this equation in powers of n, ^{we obtain}

for the n # ^{0 solution}

where

n2 vanishes at the critical temperature Tc ^defined by

as long ^as

This 2nd order transition between a ferromagnetic

and a paramagnetic phase changes ^{to first} order,

when condition (3.19) ^{ceases to be verified. This} happens ^at the tricritical point

where a new solution of equation (3.16) ^{with lower}

free energy appears.

The first order phase boundary between the P and F phases may be obtained numerically by equating

the free energies ^{of the two} phases. The result is shown in figure 2, ^{where the} point ^T

^0, Jo

^0.5

at which the first order line touches the T = 0 axis

can easily ^{be obtained} analytically. ^{We also show the}

curve labeled T, limiting metastable F solutions.

In order to obtain information about cycles ^we compute ^the parameter qo

In the P phase ^this gives

yielding cycles ^for Jo ^0.

vv

Fig. ^{2. - a = 0} plot ^of phase diagram. ^TCP indicates the tricritical end-point ^{at T}

^2/3, Jo ^{= - 1/3} log 2, marking change ^{from a} first order (continuous line) ^{to a second}

order transition.

3.3 SYMMETRIC AND ANTISYMMETRIC SOLUTIONS.

In the cases studied up ^{to now we} only ^{found the} following types of solutions :

i) symmetric solutions :

ii) antisymmetric solutions :

We believe these to be the only ^{ones, because}

solutions with n v =1= e v or p =1= r ^would correspond ^to breaking spontaneously ^the symmetry S - a ^and

since we have failed to encounter this at T

0, we

conclude that it should not occurr at all.

As for the antisymmetric solutions, they only ^exist

at T

0. This can easily ^{be seen} using ^{the relation} p ^{= r}

ql/2 ^{in the} expression ^for E 1, equations (2.28)

Which would yield complex ^order parameters.

This doesn’t happen ^{at T}

^0, because the offending

terms vanish in this limit. Thus for T> 0 only symmetric solutions exist and the order parameter equations ^{for this case} follow from equations (2.28).

Restricting ^{ourselves to retrieval states} they ^are

where

and

From these equations ^we obtain the following phases and transitions :

a) Paramagnetic phase.

Here all order parameters vanish, except qo and

ilo, ^which satisfy

For high temperatures ^this gives

Whereas for low temperatures ^{we see from}

equation (3.27a ) ^that qf -+ 1 ^{as e- / T, so that we}

may write

implying

this last result agreeing ^{with our} discussion about the

occurrence of cycles ^{at T}

0 in the symmetric phase.

b ) Transition spin-glass ^to paramagnetic.

Only ⁿ

^0, all other parameters ^{are finite.}

The equation for the 2nd order transition surface

Tg (a, ^Jo) between P and SG phase ^{is obtained} expanding equations (3.25b), (3.25c) in powers of qu

which yields together ^{with the} equations for qo and

go ^the following behaviour for Tg ( ex , J 0) ^{for a 1}

and Jo = 0:

Since we know that at T

0 this transition is first order, ^we conclude that there should exist a tricritical line (TCL) ^on the surface (3.31) separating ^these

two different critical behaviours. The TCL may be obtained numerically imposing the existence of two different solutions of equations (3.25a), (3.25c),

which coincide at the TCL, ^both having ql

^0.

Figure 3 shows the projections of the TCL onto the

planes ^T

^{0 and} Jo

^{0. In} agreement ^{with sec-} tion 3B, the line reaches the a

0 plane ^{at the} point

T

2/3 and Jo = - 1/3 log ^{2. We} furthermore ob-

serve that it exists only ^{in the} half-space Jo ^0.

The first-order transition SG-P is obtained equat- ing ^{the free} energies ^{of these two} phases.

Fig. ^3.

Projections ^{of the} tricritical line (TCL) ^{onto the} planes ^T

⁰ (broken line) ^and Jo

⁰ (continuous line).

The broken line exists only ^{in the} region Jo ^0.

c) ^Retrieval phase.

Solving numerically equations (3.25) ^{we obtain the}

surface T R ( a , 10) below which retrieval states be-

come metastable. The results is shown in figure ^4.

For Jo

^{0 we obtain a} phase diagram very similar the ^one of Hopfield’s model, except ^{that for}

a > 3.74 ^we find a paramagnetic phase with qo 0,

which ^we therefore associate with the existence of

cycles in Little’s model. In the traditional high- temperature ^P phase ^we have qo > 0.

Fig. ^4.

Jo = 0 plot ^of phase diagram. Tf is the transition temperature between F and SG phases. The continuous lines are first order transitions, while broken ones are

second order. The inset shows the appearence of the P

phase ^{at T}

^{0 and} large ^{values of a.}

d) ^The phase-diagram ^{around T}

¹ for Jo

^0.

The aim of this section is to obtain the behaviour of the curves TF ( a ), limiting the stable and metastable retrieval regions, ^{in the} vicinity ^{of the} point

T

1 and a

0, ^{where we} have discontinuous first order transitions but with small order parameters, ^so that the equation (3.25) may be expanded in powers of n and t -1 - T. We obtain

From these we get

with

Proceeding ^as in reference [3], ^{we obtain}

The behaviour of TF (a, 0) ^near T = 1 is obtained

equating ^{the free} energies of the F and SG phases, yielding

This together ^with equation (3.34) gives

where we have included the results for Hopfield’s

model in parentheses ^for comparison. Thus the F and retrieval regions ^are slightly larger ^{in Little’s}

model.

e) ^{The limit} Jo - ^00.

In this limit we obtain from equation (3.25)

with

The equations ^{reduce to the} corresponding ^ones

of Hopfield’s ^model provided ^{we rescale} T Little

2 7"Hopfic)d’ This result is reasonable considering ^that large ^values of 10 suppress cycles ^{and tend to} align

0’ i with Si. ^{We also see that the} synchronous dynamics ^{is much more stable to} noise than the

asynchronous ^{one for the same retrieval} capacity, provided ^{that neurons have a} sufficiently large ^self-

interaction 1().

4. Replica symmetry breaking.

It is well known, ^that replica symmetry breaking (RSB) is necessary ^to stabilize the spin-glass phase ^at

low temperatures. The effects of RSB in the F and SG phase ^{are similar to the ones in} Hopfield’s ^model

and thus small. For example, ^the entropy ^at T = 0, J" = 0 in the FM phase ^{is S = -1.4 x} 10- , at a = a,

0.138 and vanishing exponentially

.

In the SG phase ^at

The analog ^{of the} Almeida-Thouless line can be

computed ^as in reference [3]. ^The sign ^{of the}

replicon ^mode changes ^when

where

where we tested for instabilities only in directions

q a¡3’ ij a¡3 with a =1= f3 .

The entropy of the low temperature ^P phase ^is

also negative, going ^{to - oo as T -+ 0} exactly ^{as in}

the Sherrington-Kirkpatrick (SK) ^model [9]. ^Since

RSB has nothing ^to say about this unphysical situation, ^{we leave this matter for a} discussion in the next section.

5. Discussion.

In this paper ^we have analised the phase diagram

and storage properties ^{of a} synchronous ^{model for}

associative memory. Our results ^are qualitatively

similar to the ones of Hopfield’s model, albeit with the following differences :

a) ^{there is a new} parameter Jo, which may be used to control the occurrence of cycles

b) ^a tricritical line _appears at the surface separat- ing the P and F phases

c) ^{for or 0 we find a P} phase ^{even at T}

0,

which is not connected to the usual high temperature P phase. ^This phase ^has negative entropy, ^whose value goes ^{to - oo at} T

0 as in the SK model.

RSB is expected ^{to cure the} unphysical aspects ^of the SG and F phases, ^{but cannot exorcise} problems

in the low temperature ^P phase, ^since only qo and

go are nonzero here. Breaking ^the symmetry along

the diagonal ⁱⁿ q a¡3 ^and ij a¡3 ^is apparently ^useless,

since we found this phase ^{to be} stable in the directions qa a and 4,,a-

Let us note that this P phase ^{is our} only option ⁱⁿ

the region Jo -1, ^{where a} solution with qo 0 is to be expected. ^{The SG} phase with qo 0 doesn’t exist for ^a 2 ^{as can} be seen from equation (3.8)

or

and comments there after. RSB may extend the size of the SG solution, ^so that the P phase may ^some

how be discarded. This point ^is being reserved for a

future study.

Although ^{this P} phase ^is unimportant for retrieval purposes, for which the replica ^{scheme seems to} provide results in agreement ^with simulations, ^{we do}

not see how these diseases may be cured within the

replica scheme. We have studied this phase by

simulations at T

0. In figure ^{5 we} show the result of measuring ^the fraction q

n/N, where n is the number of spins belonging ^to cycles, against ^{N. We} interpret these results saying ^that qo (= 1 - 2 q )

never reaches the value qo = -1, ^as predicted by equation (3.6), but rather that qn - ⁺ 1 with only ^a

finite number of spins oscillating. This would exclude T

0, Jo ± ^{0 solutions with} Q ^{0 of section 3 and}

thus stigmatize ^{this P} phase ^{as an} artefact of the

replica symmetric calculation.

Fig. ^5.

Histograms ^{of the} fraction q ^of spins belonging

to cycles ^{for N}

100, 200 and 300. The parameters

«

0.5 and Jo

^{0 are} kept fixed and we average ^over initial states and realizations of the ( );’) .

Acknowledgments.

The research of R.K. is partially supported by CNPq

and J.F.F. holds ^a FAPESP fellowship. Simulations

were carried out on a VAX 780/11 computer,

partially maintained by funds from CNPq.

References

[1] ^{HOPFIELD, J.} J., Proc. Natl. Acad. Sci. USA 79 (1982) 2554 ; ^{ibid 81} (1984) ^3088.

[2] PERETTO, P., Biol. Cybern. ⁵⁰ (1984) ^51.

[3] ^{AMIT, D. J.,} GUTFREUND, ^{H. and} SOMPOLINSKY, H.,

Ann. Phvs. 173 (1987).

[4] Parallel Models of Associative Memory, ^{Eds G. E.}

Hinton and J. A. Anderson (Lawrence ^Erebaum Ass.) ^1984.

[5] ^{LITTLE, W. A.,} Math. Biosci. 19 (1974) ^101.

[6] FONTANARI, J. F. and KÖBERLE, R., Phys. ^{Rev. A 36} (1987) ^2475.

[7] ^This duplication, ^which arises very naturally, ^when

one solves this model, has also been used by ^the

authors of reference [3] and J. L. van Hemmen, Phys. ^{Rev. A 34} (1986) 3435. If the cycle length

is l, ^{we have to} introduce l copies ^of phase

space.

[8] ^See appendix ^A of reference [3] ^{for a} similar cal- culation.

[9] KIRKPATRICK, S. and SHERRINGTON, D., Phys. ^Rev.

B 17 (1978) ^4384.