Dynamics of a Neural Network Composed by two Hopfield Subnetworks Interconnected Unidirectionally

HAL Id: jpa-00247083

https://hal.archives-ouvertes.fr/jpa-00247083

Submitted on 1 Jan 1995

HAL is a multi-disciplinary open access archive for the deposit and dissemination of sci- entific research documents, whether they are pub- lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Dynamics of a Neural Network Composed by two Hopfield Subnetworks Interconnected Unidirectionally

L. Viana, C. Martínez

To cite this version:

L. Viana, C. Martínez. Dynamics of a Neural Network Composed by two Hopfield Subnetworks Interconnected Unidirectionally. Journal de Physique I, EDP Sciences, 1995, 5 (5), pp.573-580.

�10.1051/jp1:1995147�. �jpa-00247083�

Classification Physics Abstracts

05.50q 05.90m 75.10Hk

Dynamics of

_a

fleurai _{Network Composed by}

two Hopfield

Subnetworks Interconnected Unidirectionally

L. Viana

(~)

^{and C. Martinez} (~)

(~ Laboratorio de Ensenada, Instituto de Fisica, UNAM, ^A. Postal 2681, 22800 Ensenada, B.C., México

(~) ^{Centro de} Investigaciôn Cientifica _y de Educaciôn Superior de Ensenada, Programa de

Posgrado _en Fisica de Materiales, A. Postal 2681, 22800 Ensenada, B-C-, México

(Received

9 November 1994, accepted 30 January

1995)

Abstract. By _means of numerical simulations, trie dynamical behaviour of _a Neural Network composed of two Hopfield Subnetworks interconnected unidirectionally ^and updated synchron- ically

(at

zero temperature T

= 0) is studied. These connections _are such that each of the N

neurons on the first subnet sends information to exclusively _one of the N _{neurons on} the second.

A set of p patterns, composed by _a subpattern for each of the subnets, is stored according to _a Hebb-like rule. The recoverability of _one particular subpattern by the second subnet is studied

as a function of N and the load parameter _{cY =}

p/N

in the _case when the first subnetwork has already recovered its corresponding subpattern.

l. Introduction

During

the last few _years, there has been _a

growing

interest in the

study

of

Ising-like

models of Neural Networks

(NN),

that is, the

study

of the collective behaviour of _a

large

number N of

simple interacting

twc-state elements called _neurons. These systems ^have become the

prototype model for the storage and retrieval of information in _a content-addressable _{way, as} stored information acts _as attractors of the

dynamics

of the system. ^One of the most studied models _is the

Hopfield

model

[Ii,

which consists of N elements connected to each other with

symmetrical

interactions between them _as

given by

Hebb's

learning

rule _[2] for _p patterns. It has been found that in the

thermodynarnical

limit

(N

_-

cc),

for _a

=

(p/N)

_{- 0,} ail _p patterns

con be stored

exactly

_{[3]; on} the other

hand,

for _{o m} 0, the information _is stored with _a few

mistakes,

the number of which increases with _a, until for _o

= oc m 0.138, there is _a sudden

catastrophe

and information _{can no}

longer

^be retrieved [4].

In this _{paper we}

study

the

dynamical

behaviour at _zero temperature

(T

=

0)

^of _a ^finite NN

composed by

two

subnetworks,

each with

intemalsymmetric

connections between _every

pair

of _neurons

given ^by

^{the Hebb fuie. These} subnetworks _are

unidirectionally

interconnected _to

each other,

by

_means of strong ^Hebb-like connections in such _{a way} that each of the N _neurons in the first net

(subnetwork

₁₎ sends information _{to one,} and

only

_one, of the N _{neurons in} the

@

Les Editions de Physique 1995

574 JOURNAL DE PHYSIQUE I N°5

Subnetwnrkl Subnetwark2

Fig. 1. We considered two finite coupled subnetworks interconnected unidirectionally, in such _a

way that each of the N elements

in subnetwork 1 sends information _{to one,} and only _one, of the N elements in subnetwork 2.

second _one

(subnetwork 2) (see Fig. 1).

In this system, _we store p random unbiased patterns

or states

(memories) (($)

_{(1 =} 1,..

,

2N),

each

composed by

two

subpattems

_or submemories and stored in its

respective

subnet (1 = 1,..., N and

= N +1,..., 2N,

respectively).

An

appropriate

value for the connections between the two subnetworks is

expected

to allow for the recovery of _a

complete

pattern,

provided

the initial state of the first subnet is "close

enough"

to the

corresponding subpattem.

This is the issue _we address in this _paper:

by dring

computer

simulations _v~e will

study

the

recoverability

of _a

complete

pattern when subnetwork 1 has

already

recovered its

corresponding subpattem,

_{as a} function of the number of stored patterns.

Studying

this kind of models is of interest _{as we} know that

complex

behaviour is constituted

by

_{a serres} of

simpler operations.

For

example,

in

qrder

to

read,

it is

important

to

recognize

letters,

but _we need to know that _a

string

^of letters represents a word. In order to

exemplify

the

problem

to be

studied,

_we could make _an

analogy consisting

in

storing

2-letter words

(patterns),

formed

by

letters

(subpatterns).

In this way, letters would be stored in the subnets while words would be stored in the whole network. For

example,

_we could _store the word

"Si"

(pattern numberl) consisting

^{of letters} "s"

(subpattern()

and "1"

(subpattern[),

where upper index refers to pattern number and the lower _one to subnetwork number. In _a similar way, we could aise _store other 2-letter

words,

such _as "no"

(pattern

number

2).

If _we intended

to recover the word "si"

by putting

the initial _state of the first subnetwork within the basis

of attraction of the letter

"s",

_we could retrieve the word

"Si",

but

perhaps,

_we could aise retrieve the word "so" _or the letter "s"

together

with any nominated

(another letter)

_or net

ilominated

_state

(a spurious state)

_[Si. For the sake of

clanty

and without

losing

_any

generality, throughout

this paper we will

always

refer to the

retrievability

of patternl

by

subnetwork 2

(although

tue results

apply

to any of tue

patterns),

therefore

eliminating

the need of _a script

to indicate pattern ^number in the variables notation.

In this

problem,

subnetwork 1 is _a

typical Hopfield

Neural

Network,

which evolves inde-

pendently

of _any

dynamical

_process

taking place

_m subnetwork 2. The sizes of the basins of attraction for such _a NN have been calculated

by using severa1techniques,

_{SO we} will net re-

peat ^such calculations.

By dring

computer

simulations,

Forrest _[6] evaluated the _mean fraction

f(q~)

of _states which _are recalled with less than

N/16

_{errons as a} function of _q~ and _p

= a

IN,

where q~ is the

overlap

between tue initial state of tue network and tue related nominated pattern. This criterion is

equivalent

to

demanding

_a final

overlap

_qf >

7/8,

_{or a} fraction of

mistakes lower than 0.0625, to consider tue information _as

correctly

recalled. Another _group of methods used _to calculate the size of the basins of attraction for the

Hopfield NN,

in the

thermodynarnicallimit,

consists _in

iterating numencally

the flux

equations

^for the

overlaps

be- tween the _state of tue system and each of tue _p stored patterns until _a fixed

point

is reached [7];

following

this

approach,

Viana and Coolen _[8]

calculated,

for tue Hebb

model,

tue fraction of ail initial states which _are within _any of tue

(p

₌ 1,.

.,18)

basins of attraction related to tue

nominated patterns.

By dividing

such fraction

by

_{p, we con} obtain tue _mean fraction of ail initial states which lie within tue basm of attraction of _one

(any)

of tue patterns.

Knowing

these

previous results,

_{we can} concentrate herein _on

studying

tue behaviour of subnetwork 2

provided

^subnetwork 1 bas

already

recovered tue information related _{to it.} We would like to mention that there is another _group

working

_on

coupled

networks from _a dilferent

perspective: they

_are

modelling

semantic memory

by considenng

tue _case of biased patterns stored _on several

coupled

subnetworks with diluted connections between them [9].

2. Trie Model

We consider _a Neural Network

composed by

two

subnetworks,

each constituted

by

N

interacting

twc-state _neurons with

symmetric

interactions. Bath nets _are connected

unidirectionally

in

such _a way that i~~ _neuron in subnetwork 1 sends information

exclusively

to tue

(1+ N)~~

neuron m subnetwork 2. Tue

dynarnical

evolution _is assumed to take

place synchronically

at

zero

(T

=

0)

_noise;

^therefore,

^these two subnetworks evolve in time

according

to tue

updating

rule

Si(t +1)

=

Sign(ht(t)), (1)

where

S~(t)

₌ +1 denotes tue

dynamical

state of tue i~~ _neuron at time t with

= 1,..

,

N

corresponding

to subnetwork 1 and

= N ₊ 1,.

,

2N to subnetwork 2;

h)

is tue field at site

on the k~~

subnetwork,

due to the interaction with the rest of the _neurons. For the first network this field is

given by

N

hi (t)

=

L _{Àjsj, (2a)}

with = 1,.

,

N,

and for the second network

hÎ(t)- f

j ~

~

v

j+£w~

s ~

~ ~~~ ~~~

~ ~ ~~~~~

(2b)

with

= N +1,. ,2N. The interactions J~j =

Jj~

within each subnetwork _are such that

the first N components of each of _p random unbiased patterns

(($), (($

= +1 with

equal probability),

bave been stored in subnetwork 1, and the

remaining

N components ^{in subnetwork} 2; these connections _are

given by

j

f

(M(M

(3~)

lJ Jf i j'

p=i for1

# j

^{and within tue} same

subnet,

and Jm

= 0. W~j ^is an

asymmetrica1Hebb-like

connection

sending

information

unidirectionally

from

spin j

in subnetwork to

spm1

in subnetwork 2, and is

given by

P

Wq

"

£ ($()ôi,j+N, (3b)

p=1

576 JOURNAL DE PHYSIQUE I N°5

for

j

₌ 1,..., N and

= N +1,..., 2N. In tue last

expression

tue normalization _was chosen in _a way that subnetworks do not

get

disconnected _as N grows.

In order to characterize how close tue system is, at any time, to tue pattern to be

recalled,

we define m~ _{to be tue}

overlap

between tue

dynamical

state of subnetwork k

(S~)

and its

corresponding subpattern( (()), given by

~k jl

~

(~)

N ^{1 ~'}

i=(k-1)N+1

with k ₌ 1,2 for the first and second subnets. Tue _use of tue

subscripts (f)

and

(o),

in

m)

and

ml,

will account for final and initial

values, respectively.

Notice that the maximum value for the

overlaps

has been normalized to 1 within the

subnets,

_so the fraction of correct bits recovered

by

each of the subnetworks is

given by ) (1+ _{m~). Therefore,}

the

overlaps (m~)

are related to the total

overlap

_q, between the

complete patternl

and the state of the whole system, as follows

~

11

~

~~ ~~ ~~~ ^~

~~Î

3. Numerical Simulations

We

study

the

dynamical

behaviour of the network for

synchronous updating

of the

spin

states

as

given by equations (1-2).

As _we mentioned

before,

_{we are} interested in

evaluating

^the "size"

of the basin of attraction of _a

subpattern, provided

subnetwork 1 has

already

recovered its

corresponding

part. ^{We considered values of N}

ranging

between 32 and

1024,

^{and several} values of _p, the

highest

of them

being

144 for the

largest

network

(N

=

1024).

For each set of values _p and N

considered,

_we studied _a collection of 100 dilferent neural networks where the patterns were stored

according

to the

learning

rules

given by

equations

(3).

In ail _cases, the initial state of subnetwork 1 _was set to be Si

=

(),

for1 ₌ 1,.

,N (ml

₊

_1),

while

various initial states for subnetwork 2 _were

considered;

_{so we} knew subnetwork 1 _was at, or at least _very close to, ^{tue minimum related to the}

subpattern.

After tue whole network reached

a

steady

state, the criterion used to

identify

whether _or net it had recovered it, _was

having

_a

final overlap

mi

with this memory of order 1, while the absolute value of the other

overlaps being significantly

lower

(of

order

1/@).

The first

thing

_we noticed _was that the basm of attraction related to the

subpattern,

if subnetwork has

already

recovered its part, is

huge (much bigger

thon the size it would have for _a

Hopfield

network with the _same N and _a and without _an

externa1signal);

except for _a

small

a-dependent

fraction of _cases, shown _m

Figure

2, the

following happens:

If subnetwork is put in _an initial state

given by

_S~

=

fi

for1

= 1,.

..,

N,

_so

m(

= 1, then the

persistent signal coming

from subnetwork 1 drives subnetwork 2 towards the minimum related to the

subpattern, independently

of the value of

m(. Furthermore,

the value of the final

overlap mi

is

generally

the _same for _any initial state of subnetwork 2

(having

dilferent values for dilferent realizations

(J~j)).

Crosses _xx in

Figure

3 show _a typical relationship between tue initial and final

overlaps

for _an

arbitrary

network with N

= 1024 and _p

= 127. Notice that

even for _an initial state with

m(

= -1 tue system evolves towards tue _same final _{state as it} does when values

m(

_{m +1 are} used. Similar results _were found for most N and _p

considered,

even if the initial state of subnetwork 2 _was identical _to other nominated pattern;

however,

_we

found _a few isolated _cases difficult _to

quantify (always

for _{even p,} and the lowest values of N

ioo

go ,

* so

lu

8o

# 50

40 30

20

tu o

o ce

dilferent (J~j) for each p value,

x x x x x x x x x _x x x x

x

x x x _x

o o o

o

o o _o o o

"-

o o o

E

~ o o o o o o o

o.2 -1

,~

Fig.

cases: (xx)

represent the

with

ecovered its depends on m(, is by _iamonds (oo) for a case with _N = 64 _and

p =

4.

578 JOURNAL DE PHYSIQUE I N°5

l~Ù1 li=6

2

o

8

<00z

6 p=<2

?

o

8

992

6 p=<8

;

2

o

0

972

6 p=24

?

~0 _{0.2 o-Î o-G 0 8}

~>

J

Fig. 4. Statistical study _over 100 realizations in each case, of the shift in the positions of the attractors, for N

= 1024 and various

even values of p. The figure _in the right _{upper corner} indicates the percentage of _cases in which subnetwork 2 did not recover the pattern. For odd p, there _is

a similar behaviour, but the shift is bigger,

as con be _seen in Figure 5.

considered)

where, even when the information _was still

recovered,

its

quality depended

_on the initial

overlap m(.

This is illustrated for _a

particular

_case with N

= 64 and p = 4

by

diamonds

(c<)

in

Figure

3.

These above-mentioned results _are

interesting,

_as

they imply that,

if

ml

= 1, _a

particular

network is able to recall _a

complete

_{pattern, or} it is trot, with _an

a-dependent probability.

But if this

happens,

then the basin of attraction _covers the whole _space of states, since any initial

state for subnetwork 2 will evolve towards _a minimum related to the

subpattern. However,

as can be _seen in

Figure

4, ^the

positions

of these minima _are

considerably displaced

from the nominated patterns

(as m)

_{< 1)} which reflects _a reduction in

the'quality

of the information recovered.

Figure

4 shows _a statistical

study

of the shift in the positions ^{of the minima}

(mi),

for N =

1024,

and various values of _p

(o).

These

graphs

show the percentage ^{of times} i~ in which

m)

_was obtained _as the final

overlap,

with respect to the total of _cases in which the

subpattern

was recalled. The

figure

in the

right

_{upper corner} indicates the percentage ^of _cases in which the

subpattern

was not

recovered;

in ail those _cases, subnetwork 2 _never reached _a

steady

state but

kept switching,

due _to frustration, _among several states with the _same value

mi.

It is

interesting

to note that the behaviour of this model

depends

_on the parity of _p: for _even values the percentage ^{of times the} information _is recovered is

slightly

smaller thon it is for odd values;

however the

quahty

of the

information,

when

recovered,

is better for _{even p} thon it is for odd _p

(Fig. 5).

As _a increases, the

quality

^of the recalled information diminishes and the shift in the

mean value <

mi

> grows; this elfect is stronger ^{for small} p

values,

and becomes smoother _as p grows. It is not

possible

to

give

a cntical value for _o, for finite

N,

_smce these elfects increase

09

os

oi

08 A

~, ~~

V 04

~~

02

_,~~

01

0

o 002 004 008 oo8 O.l o.12 o.14

7.

^{5.-Mean value of the final overlap}ce ^<

^m)

^{> with bars} representing _{a =}

<

(m))2

> <

(m))

>~ for N ₌ 1024 and _p taking values between 3 and 144. A dotted

(solid)

bue

conuects _even

(odd)

values for _p.

gradually.

4. Discussion

It is very

important

to

emphasize

that if the initial state of subnetwork is within the basin of attraction of _one pattern, ^subnetwork 2 will evolve towards the attractor related to that _same pattern with _a

high a-dependent probability [loi. Therefore,

in these cases, neither other memory nominated _{or net} will

play

_a rote in the _recovery of information

by

subnetwork 2,

as tue basin of attraction _covers tue whole space of states. This is very dilferent from what

happens

in _a usual

Hopfield model,

where tue _average size of tue basins of _attraction of each of tue _p stored memories is tue same, so tue fraction of _states

leading

to each of them diminishes

as p grows.

In these two

coupled networks,

tue

strength

of tue

incoming signais

in subnetwork 2 is _so

important,

that it induces tue latter into tue minimum

required,

_no matter its initial state.

This

finding

is consistent with tue observation _{that nervous} systems ^show a dilferent _set of

activity

patterns when observed in

freely moving

animais than when observed _in vitro, ^where many

important

^alferent

inputs

_are eliminated

Ill].

It aise

explains

^tue

repertoire

^{of dilferent}

collective _responses, of _{a group} of

interacting

_neurons,

depending

_on the afferent

signais.

It is difficult _{to compare} the load

capacity

^of this system with that of the

Hopfield

^model _oc m

o.138 for N

- cc, since _many dilferent criteria _cari be used to make such _an

analysis.

Notice that _as subnetwork 1 is _an N-element

Hopfield Network,

_we carnet store _more

subpatterns (and

therefore _more

patterns)

than _{p m} acN;

therefore,

tue relation between tue number _p of

complete

patterns ^{and tue total number 2N of units}

gives

_{us as} a storage ^limit of tue

complete

network half that of tue _common

Hopfield

Network.

However,

in order to _recover

information,

we aise need half the percentage ^{of bits}

required

to _recover it in _a

Hopfield

NN. For it is

only

necessary for the initial state of subnetwork 1 to be within the basin of attraction of _a

given

subpattern, independently

of the initial state of subnetwork 2

(it

could _even coincide with _any

580 JOURNAL DE PHYSIQUE I N°5

of the other

subpatterns).

On the other

hard,

each of the

2p subpatterns

could be considered

as a pattern

itself,

_so

by taking

this

position,

for

large

N this system ^would

equalthe capacity

limit of the

Hopfield network,

but with half the total number of

connections,

_as these would be 2N~ _{for a}

Hopfield

NN with 2N elements and N~ _{in this case; but}

we would still need

only

ualf the bits which _{are necessary} to _recover eacu of the _p patterns in the usual

Hopfield

model.

Tue results tuat _we found for this system

composed by

two finite

coupled

subnetworks

are very

promising. Tuey

could be useful to

implement

cuains of

coupled

networks

storing

information

composed

of

meaningful subsets,

wuicu _{are to} be recalled

sequent1ally

in time and space. It is

important

to

point

out tuat in tuis

model,

tue fraction of the total information needed _to make the recovery

possible

_is much lower than in the

usua1Hopfield model,

_since tue

persistent signal coming

from tue previous subnetwork when it has reached _a stable

state acts as a turesuold _on tue _neurons of tue second subnetwork

drawing collectively

its

dynamic

state into tue basin of attraction related to tue stored _{memory in}

consideration,

tuus

playing

_a rote similar to tuat of _an "init1al state".

However,

_one of tue main

problems

found

is tuat information recalled

by

tue second subnetwork bas many errors witu respect to tue stored information

(low mi values).

Tuis suggests to us, as a next step, to

study

_a

dynamical

process

composed by

_{as many} stages as subnetworks _are

coupled, wuere,

alter subnetwork _n reacues _a

steady

state, ^tue

signal coming

from subnetwork

(n

₁₎ is

suppressed.

In tuis _way,

subnetwork _n would be allowed _to searcu for _{a new minimum}

taking

_{as an} initial _state tue

one obtained wuen still connected to its

feeding

partner. ^Tuis

possibility

is

currently

under

investigation

_[12].

Acknowledgments

Tue autuors wisu _to tuank Dr.

Miguel

Virasoro for

uaving suggested

_us to

study

_a cuain of

networks,

and M.C. Armando

Reyes

for

computing

support. Tuis work _was

partially supported

by

DGAPA

Project

No. 1N100294 of tue National

University

of Mexico

(UNAM),

and _a

CONACYÎ scuolarsuip

for C.M. Numerical simulations _were

performed

_on UNAM'S

Cray

Y-MP

4/432

supercomputer.

References

[1] Hopfield J-J-, Froc. Nat. Acad. Soi. USA 79

(1949)

2554.

[2] ^Hebb D.O., The Organization of Behaviour

(Wiley,

New York, 1949).

[3] ^Amit D.J., Gutfreund H. and Sompolinsky H., Phys. Reu. A32

(1985)

1007.

[4] ^{This value for} oc was calculated within the rephca symmetric approximation, _m the limit N _{- ce.}

[Si ^{One word of caution about this} analogy must be said: the analogy has its pitfalls, when used for the visual representation of letters, since here _{we are} considering non-correlated patterns with

zero mean and without any spatial structure.

[fil Forrest B-M-, J. Phys. A21

(1988)

245.

[7] Coolen A.C.C. and

Ruijgrok'Th.W.,

Phys. Reu. A38

(1988)

4253.

[8] Viana L. and Coolen A.C.C, J. Phys. I France 3

(1993)

777.

(9] ^Virasoro M.A., Reich S. and Laurogrotto R., to be published.

[10] ^This o-dependent probability has its origm in the cross-talk _noise.

[Il]

Johnson S-W-, Seutin V. and North R-A-, Science 258

(1992)

665.

[12] ^Viana L., _in preparation.