Adaptive architectures for Hebbian network models

HAL Id: jpa-00246522

https://hal.archives-ouvertes.fr/jpa-00246522

Submitted on 1 Jan 1992

HAL is a multi-disciplinary open access archive for the deposit and dissemination of sci- entific research documents, whether they are pub- lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Adaptive architectures for Hebbian network models

Karl Kürten

To cite this version:

Karl Kürten. Adaptive architectures for Hebbian network models. Journal de Physique I, EDP

Sciences, 1992, 2 (5), pp.615-624. �10.1051/jp1:1992105�. �jpa-00246522�

J. Phys. I France 2

(1992)

615-624 MAY1992, PAGE 615

Classification Physics Abstracts

87.30 75.10H 64.60

Adaptive architectures for Hebbian network models

Karl E. Kfirten

Institut ffir Neuroinformatik, Ruhr-Universitit Bochum, D-4630 Bochum, Germany Institut fur Theoretische Physik, Johannes-Kepler-Universitit Linz, A-4040 Linz, Austria

(Received

13 December 1991, accepted in final form 24

January1992)

Abstract. We present a novel procedure which allows _a neural network to evolve to _a

quasi-optimal connectivity such that information to be memorized is engraved _as efficiently

as possible. The emerging connectivity structure of the resulting partially connected network

depends strongly on the information the network is asked to memorize. A stability _measure of the quality of storage _{as a} function of the degree of the connectivity is shown to attain _a

maximum value, which lies substantially above the stability of the traditional fully connected system.

I Introduction.

There is _no doubt that artificial neural networks which _are

supposed

to

perform intelligent

tasks have to be

designed

with great care and considerable

ingenuity. Moreover, tayloring

network architectures

according

to

context-dependent

^tasks is _a

problem

of fundamental

importance.

Nevertheless,

most of the

currently popular

models _are based _on fixed network architectures

or even full

connectivity

and lack basic

adaption properties

present in

living

networks. Hence, these models cannot

adapt specific

structures matched to the task the network is asked to

perform.

In contrast, variable

connectivity

for each individual _neuron

during

the

learning

session allows the network to

undergo

_an

optimization

_process, where

optimal

and _even minimal

connectivity

structures

might

_emerge. Moreover, flexible network

topologies

allow network

designs

_or

learning strategies aiming

at

economizing

on the number of connections

highly

matched to the information the network is asked to process.

2. The

partially

connected network model.

Our model system consists of N

binary

formal _neurons which _assume either the value _{a; =} -I when _neuron I is "silent" _{or a;}

= +I when it "fires". In contrast to

Hopfield's

model

ii]

our

system is not

fully

connected and each _neuron I receives

input

from

only

It; other units of

the

network,

which renders the

topology

^flexible. The

firing function, h;, representing

the net internal

input

to _neuron I at time t is then modelled

by

^the

commonly

used linear

superposition

of

weighted input signals

h'(I)

_"

LCiJ'J(I)/

II Cl l12

(2.I)

(I) with the

spherical

normalization

II Ci

((2" ~

Cl;

(2.2)

~

The _sum

£~;~

^is

^only

^taken over those

K;

_neurons that interact with _neuron

I,

while self-

coupling

efsects _are excluded. The state of the network at time t is then evaluated

according

to the deterministic threshold rule

«;(t

+

I)

₌

Sgn(h;(t))

I

= I,

,

N.

(2.3)

Note that the

parallel

_or

synchronous updating

in

(2.3)

_can be

replaced by

_a

sequential

_or

asynchronous dynamics

_as in the

Hopfield

model

iii.

When _a set of _p

= aN

arbitrary

_memory

configurations presented

as n-Aimensional

firing

patterns

S~,...,Sl'

_E

(- I, 1)~

are to be encoded in the

network,

the

oft-diagonal coupling

coefficients _can be defined via the Hebbian

prescription

c(

p

=

£ _{SfS) (2.4)}

p=1

usually

called outer

product rule,

_or

by

the

corresponding "clipped" coupling

coefficients with

cj

=

sgn(c( (2.5)

Note also that due _to the

incomplete connectivity,

the

synaptic couplings

_are not

necessarily

correlated for different cells and hence _are in

general

_no

longer symmetric.

Ideally,

under the

learning

^rule

(2A)

and

(2.5)

all the

prescribed

patterns ^of information should become stable fixed

points

of the

dynamics. However,

it is well known that for

fully

connected networks the choices

(2.4)

and

(2.5)

allow faithful storage ^{of the} information

only

up to _a critical

storage capacity

_ac

= 0,14 and _ac

=

0.102, respectively

_[2].

By

contrast, we

will _see that for

networks,

where each cell chooses its K;

neighbours

in _an

optimal

_way,

perfect

information

storage

^is

possible

far

beyond

these critical

capacities.

3.

Stabflity.

A reliable _measure of the

quality

of memorization is

given by

the

magnitude

of the

stability

parameters _K;p ^defined as

K;» =

Sfh;

=

St L _c;;SJ/

II C; ll~

(3.i)

u)

In order to store the

prescribed

patterns

S~,...,Sl'

_as fixed

points

of the

dynamics (2.3)

it is necessary that all

stability

parameters _K;~ be

nonnegative. Furthermore,

their

magnitude

is

required

to be

as

large

_as

possible

in order to _ensure

reasonably large

basins of attraction for

N°5 ADAPTIVE ARCHITECTURES FOR HEBBIAN NETWORK MODELS 617

the individual patterns. ^{In order} to

gain

_more

insight

into the

origin ofstability

_we

decompose

K;~, defined for _a

fully

connected

network,

into its individual contributions

nj(~

=

Sfc;jS)/

ii c; [[2

I

₌

i,.. ,N (3.2)

and define the _{p x}

(N I) stability

matrix

~,

(,(j))j"I,...,N

^j#I _~~~~

' ip p=I,...,p

Here,

the matrix element Kj(~

specifies

the

stability

contribution for the I-th component ^of pattern _~ ^due to the

synaptic

connection from

j.

Note that

summing

_{over an}

arbitrary

_row

j gives merely

£

K)(~ ⁼ K;~

,

(3.4)

(I) whereas

summing

_{over a} ^column _~

produces

f

_cj(~

=

~c;jc( /

ii c; [[2

(3.5)

P ~_i P

The _sum rule

(3.5) implies

that if and

only

if the actual

couplings

_{c;j are} chosen

according

to the Hebbian

sign

constraint

sgn(c;>)

₌

Sgn(c() (3.6)

the arithmetic

stability

_mean

4~~ represented by

the left hand side of

equation (3.5)

is strictly

positive,

whereas _an anti-Hebbian

sign

restriction

sgn(c;j

₌

-sgn(c(

would lead to _a

negative

value. This demonstrates

convincingly why

Koehler's and Widmaier's

procedure

_[3]

,

based

on

couplings

with the anti-Hebbian

sign restriction,

could _not stabilize _a

single

pattern ^of information.

Equation (3.5)

then

implies

that the arithmetic _mean of the whole set of the

stability

_pa-

rameters Kj(~ taken _over all patterns _~ and all

incoming

connections

j

_can

always

be written in terms of _a normalized _{sum over} the

products

of the actual and the Hebbian

couplings

11 =

L f

_c)[~

=

(L _c;Jc()/

II C; ll~

(3.7)

u) »=i u)

Hence,

a substantial

portion

of the

couplings

must

obey

the Hebbian

sign

constraint in order to allow for _a

positive

_mean. Note

however,

that _an absolute

stability

_measure of the embedded

information is

given by

the value of the minimal ~c;~. ^In the _case of Hebbian

couplings given by equation (2.4), expression (3.2)

reduces to

ii ₌

)~i

= II

cl

l12

(3.8)

u)

whereas for

"clipped"

Hebbian

couplings specified by equation (2.5), expression (3.7)

takes the form

~ ~

'~'

~~ ~

_~~'~

~~

^{~ ~ ~~ ~~ ~~}

In

judging

the

quality

of memorization _one

might

also consider the number of

positive

and

negative

^entries in column

j

^of the

stability

matrix K; defined in

(3.3), n(

and

n[, respectively.

These numbers

obey

the relations

n(

=

lp

+

c(sgn(c;j

_)] and

nj

= §~

c(sgn(c;j

_)]

(3.10)

2 2

If the

coupling

_{c;j is} chosen under the Hebbian

sign

constraint

(3.6), n(

_assumes its minimum value

)

for

c(

₌ 0 and reaches its maximum value _p for

[c([

= p, where

[c([

^{takes its maximal}

value.

Thus,

the fraction of

positive stability

entries in _an

arbitrary

column

j

is

always larger

than and increases

linearly

with

increasing magnitude

of the Hebbian

coupling. Accordingly,

as is

the

case in

biological networks,

it may be inferred that

large

Hebbian efficiencies

play

_a

crucial role in

optimized

network models.

4. Dilution of the network

connectivity.

Basically,

there _are two

completely difserent, though equivalent approaches

for

reducing

the number of

weights

of _a

fully

connected network without

significantly degrading

the

performance

of the network.

Moreover,

_{even a} substantial

improvement

of the

performance

_can be achieved

provided

that the

coupling

coefficients have _not been chosen

"optimal".

One

might

start from

a

fully

connected network and

selectively

delete those

weights

which disfavour the stabilization of the information to be

engraved.

This

approach

has the serious drawback

that,

before the

dilution _process starts, the whole _set of

couplings

for _a

fully

connected network has to be determined. In

fact,

this

might

be _an unfeasible

task,

if nonlocal

learning

rules _are taken into account.

By

contrast, _{we propose} to start with _a

completely

disconnected network and

selectively

add

only

those connections which _are most effective in

increasing

a suitable cost

function,

for

example

the smallest

stability

parameter _~ci~ in

(3.I).

Let _us first summarize two

straightforward approaches

in section 4,I and

4.2,

before _we propose a

highly

information-

specific

and _more efficient

algorithm

in section 4.3.

4.I RANDOM _DILUTION. The

simplest strategy

^for

reducing

the number of bonds is _to

delete bonds _at random until the number of _synapses for _neuron I is reduced _to

It;.

The

dynamics

of _a

randomly

diluted and hence

asymmetric

version of the

Hopfield

model _can be solved

exactly

in the

high-dilution

limit [4], or more

precisely

under the condition It _m

logN.

It has been shown that the critical number of unbiased patterns _pc ^{the network is able} to store is _pc

=

~

K.

However,

for _an extensive number of

patterns

no

perfect

retrieval is

possible, although~here

exist clouds of attractors _{near a} stored pattern.

Furthermore,

random dilution of _a

macroscopic

fraction

synaptic

bonds _can also be shown to be

equivalent

to

adding

_an

independent

Gaussian noise to

t~e strength

of the

coupling

coefficients.

4.2 DILUTION OF THE SMALLEST BONDS. Just

eliminating

those connections with the

smallest efficacies _as

proposed by Morgenstem

_[8],

might

be

quite efficient,

but

hardly optimal.

Equation (3.8)

^reveals that for fixed

K;

this choice does not

optimize

^the minimal K;~ but the arithmetic _mean k;. The

price

for such _a

straightforward

search _may be _a

large variance,

whereas _a

near-optimal

network structure

requires

_a subtle tradeoff between _a

larye

_average k;

and _a small variance. Note that k; reaches its maximum value for _a

fully

connected network.

Apparently,

this

quantity

decreases with

increasing degree

of dilution of the weakest

bonds;

therefore, by cutting only those,

the

stability

cannot be

improved

at all.

N°5 ADAPTIVE ARCHITECTURES FOR HEBBIAN NETWORK MODELS 619

4. 3 SELECTIVE _DILUTION.

Conhidering

the

connectivity

parameter K; _as

fixed,

the

design

of the network _can be

thought

of _{as a} combinatorial

optimization problem,

where each cell I has

(~~)

_{different ways to choose its}

interacting neighbours. Ideally,

one would have to select

j/;

_{columns of the}

_stability

_{matrix defined in}

_{equation (3.8)}

_{such that the minimal}

stability

parameter defined in

equation (3.6)

attains its maximum value. Once _an

optimal

set of connections has been found for each K;

(I

< K; < N

I),

the

quantity

It; _can be used

as a variational parameter in order to determine the

optimal

number of connections which maximize the minimal

stability

parameter _K;~. The result is _a network with greatest

stability

and minimal _resources with respect to the

interconnectivity

structure.

5. Deterministic construction of the network architecture.

We _are

mainly

^interested in

sparsely

connected

networks,

where the

efficiency

of information

storage

per synapse is

usually

^much

higher

than in their

fully

^connected counterparts.

Hence,

it is _our intention here to build _a network connection

by

connection. Since the search _process is

independent

for each individual

cell,

_we will describe the

algorithm

for _a fixed cell I.

In order _to attack the combinatorial

optimization problem

_we

adapt

Branch-and-Bound

(BILB) algorithms

which have been

widely applied

to the solution of

optimization problems

of

high complexity

[9]. ^{A BILB}

algorithm

_can be considered _{as a}

general

_strategy to search for

optimal

solutions in _a

combinatorially large configuration

_space, in _{our case,} the _space of all

possible

connectivities. This strategy avoids _an

explicit

enumeration of all

possible

candidates

by restricting

the search to certain

subspaces

of the whole

connectivity

_space. The exclusion of those classes which do _not

yield near-optimal

_answers from further

competition

is based _upon the

computation

of _a lower bound of _a suitable cost function. To be

precise,

_a B&B

algorithm

consists of three components: _a

branching scheme,

_a

bounding

function and _a search strategy.

The

recipe

for

building

_our network for _a fixed _neuron I is _a5 follows. We first _rearrange N I

possible input

cells

j

in

sequential

order

according

to the number of

positive

entries

n(

defined in

(3.10). (Note

that for Hebbian

couplings

_as defined in

(2A)

and

(2.5)

this rearrangement

coincides with

ordering

the cells

according

to the

magnitudes

of their

synaptic weights c;j.)

This ordered set of candidates

(aj~,

aj~,...,aj~-

II

will be called _our first

working pool P(I).

Accordingly,

cell

ji

^{which has the maximal number} of

positive

entries

[n(~

[,

provides

the _very first connection to cell I. The

correspinding

first lower bound

b(I)

for _our BILB-like

algorithm

is then

specified by

the value of the smallest

stability parameter min(K;~)

^{calculated via}

(3,I),

P

where _neuron I receives

only

_one

single input

from _neuron ii ^{such that}

b(i)

=

njn(K;»)

⁼

r§in(SfS[) ^(5.i)

Notice that the first lower bound

b(I)

will

usually

take the value I, unless there _are maximal correlations between the I-th and the

ji-th pixels

_among the patterns such that [c;j~ = p and

b(I)

takes the value I. The second _neuron which will be connected to neuron I is chosen _as follows: Neuron

j2

of _our current

working pool P(I),

_now

consisting

of

(N 2) units,

_serves

as a trial candidate.

According

to

(3.I)

_we calculate the trial

stability

measured

by

K~R~j = rrd~

~l(C;J, S(

+

c;j~S()

~

~ ^(5.2)

corresponding

to two

incoming

connections from _neuron

ji

and

j2.

Candidate

j2

is

accepted only

if

K~RaJ >

b(1) (5.3)

such that the trial

stability

_measure

equals

_or exceeds the value of the lower bound

b(I).

In

case neuron

j2

is

eligible b(I)

is

replaced by

the

corresponding

value of _K,r;ai, otherwise

b(I)

remains

unchanged.

The search strategy continues with _{a sweep}

through

the

remaining

N 3 connections of the

working

pool

P(I

in order to increase further the

stability

of the information the network is to memorize.

Hence,

_we accept

only

those _neurons, whose trial stabilities

equal

or exceed the current lower bound

b(I).

In other

words, only

_neurons which

provide

for _an increase of the minimal

stability

will be

eligible.

Let _{us now assume} that the first _sweep

provides

_neuron I with _ni connections from _neurons

(ki, k2, ...,kn~)

such that the _new

working pool P(2)

consists of N I _ni candidates. In

analogy

to the first step _we first accept the

neuron with the maximal number of

positive entriesj

regardless

of the

stability

test

(5.3).

The second lower bound

b(2)

^{is then}

specified

with the aid of

(3.I),

where _neuron I

now receives _{ni + I}

inputs

from _neuron

ki, k2,..

,

kn~+i

km+i

T

b(2)

=

ruin(K;p)

=

min[S,f £ c;iSf]/ £ c[ (5.4)

" "

I=ki I=ki

The search strategy ^{then continues with} a second _sweep

through

the

working pool P(2)

which

now consists of N 2 _ni

remaining

candidates.

In _a I-th step, we consider

only

candidates of the

working pool P(I).

In

analogy

to the former steps we first accept the first cell of the ordered set

P(I)

and determine the _new lower bound

b(I).

This

quantity

_{serves as} acceptance criterion for

adding

further connections

during

the I-th _sweep

through

the

remaining

candidates of

P(I). Again, only

those connections _are

accepted

which do not decrease the current lower bond

b(I)

but increase the minimal

stability

measure.

Thus,

after the I-th _{sweep, we can} eliminate another _ni connections _so that the _new

working pool P(I

+

I)

consists of N

£(_~

_n; candidates.

This

fully

deterministic BILB-like

procedure

is

repeated

until the

working pool

is empty. ^It ends after at most

(N I) (weeps,

while the

learning

time is at most of the order of

tJ(N~).

Eventually,

_we arrive at _a

complete stability

spectrum _{a5 a} function of the

connectivity

and thus _can determine those

K;

connections which maximize the smallest

stability

parameter _K;p.

However,

it should be

stressed,

that _our

algorithm

should be considered _{a5 a}

working

tool for

finding

_a

"good"

^{solution in}

polynomial

time rather than _a

prescription

for the

"optimal"

solution,

which

might require

_a number of steps

exponentially increasing

with the number of

neurons.

6

Computer

simulations.

6.I PROCESSING RANDOM PATTERNS. In this section _we demonstrate how the

perfor-

mance of the network varies with

varying connectivity

for three different

strategies

of dilution:

random

dilution,

dilution of the weakest bonds and selective dilution of the

coupling

coeffi- cients _as described in the

previous

^section. The simulations have been

performed

for N ₌

100,

N ₌ 400 and N ₌ 800 cells.

Though

we observe _some effects of size

dependence

at both _ex- tremes of the

degree

of

dilution~

the

qualitative findings

of _our

study

_are not affected.

Figure

I shows the minimal

stability

parameter _{Km;n =} min K;p,

averaged

_{over some} 50

specimen

nets P

a5 a function of the

degree

of the

connectivity.

The networks consht of N ₌ 100

cells,

while the

capacity

parameter _a has been chosen _{as a}

= 0,14. Before the retrieval

phase

the patterns

are

degraded by

random noise and

presented

_a5 initial conditions for the network to recall. As to be

expected~

the simulations for random dilution of the

couplings

show _a linear decrease

N°5 ADAPTIVE ARCHITECTURES FOR HEBBIAN NETWORK MODELS 621

2.0

o o

~ l.5 o ° °

~ o ~

2

~

l-O ~

~C

~ ~ ~ ~ * *

~ii ₅ ^~

n

~

_~ ^*

~C

~

°

Z£ ° ^*

~

fi

n

_5 ⁿ

n

~~ '°

o 2 4 6 8 1, o

CONNECTIVITY

Fig. I. Average minimal stability parameter _{Kmin as a} function of the dilution for random dilution

(squares),

dilution of the weakest bonds

(stars)

and selective dilution

(diamonds).

of _Kmin, whereas

deleting

bonds with the smallest efficacies

only

leads to _a

slight

decrease _up to almost 60$l dilution. We observe that the behaviour of _Kmin is

quite

reminiscent of recent

exact results for the

capacity

_[6].

In the _case of selective dilution

following

the scheme of section

3,

_we observe that _Kmin is _a

sharply increasing

function at small

connectivities, reaching

_a maximum value

slightly

above 50$l. With further increase of

connectivity,

_Kmin shows _a slow decrease until it reaches the

limiting

value

corresponding

to the

stability

_measure for _a

fully

connected

Hopfield

network.

Figure

2

depicts

_Kmin for _a

= 0A in the _case of networks with

"optimal" connectivity.

The

figure

illustrates

clearly that,

_over almost the whole

connectivity

_range,

coupling

coefficients

"clipped" before

and not

ajier

the

learning

_process result in _a better

stability

than ^"

genuine"

Hebbian

coupling

coefficients. Since

clipped couplings

_carry less information in

fully

connected

networks,

_Kmin takes _a lower value for _very

high

connectivites.

Preliminary

work shows that _a

slightly

modified version of _our

neighbour

search

technique

can also be

applied

to the

problem

^of ternary

coupling

coefficients c;j ^E

(0,1, -1)

not restricted

by equation (2.5). Here,

each cell has

exactly (~((~))

different _ways to build its

connectivity

structure.

Figure

2 suggests ^{also that for} sparse

coniectivity,

networks with

"clipped" couplings perform

_as ^well _as ^the more

general

_ternary

choice,

whereas with

increasing connectivity

_a substantial part of the _nonzero ternary

couplings

do _not

obey

the Hebbian

sign

restriction

equation (2.5)

any more.

6. 2 PROCESSING ARTIFICIALLY STRUCTURED INFORMATION. Correlations _among differ-

ent patterns turn out to be _a serious obstacle for _a

satisfactory performance

of the

fully

_con- nected

Hopfield

model. This is not

surprising,

since the Hebb rule treats the individual patterns

equally

and considers each pattern as a new

piece

of information.

Thus,

there is _a

tendency

to enhance

overlapping

parts of information. On the other

hand, important

details which dis-

tinguish

the patterns _are not well

captured by

the network. This failure _as well _as massive interference erects result in the _appearance of _numerous

spurious

attractors which deteriorate the network

performance considerably.

6

~ ^~ ,

i

^° ° o

~j

2 I _n °

[

~

0

~ n u

f~

0 o

u

*

E- ~ n

W -2

_q n *

< -4 a

Z£

-6

11

~

-l.0

0 2 .4 .6 .8 1-O

CONNECTIVITY

Fig. 2. Average ^minimal stability parameter _{Kmin as a} function of the dilution for Hebbiari

(squares),

"clipped"

(stars)

and ternary coupling coefficients

(diamonds).

In real life

applications

stored information is _never

completely

random but contains _corre- lated structures. For

example,

there _are

large

correlations _among the 26 letters ofthe

English alphabet

and it is well known that _a network model with

couplings

based _on the

outer-product

rule

(2.5)

is not able to treat this

problem adequately

in the absence of

high-order

interactions

ill]. However, figure

4 demonstrates that _our model is _not restricted _to capture ^{random infor-} mation.

Moreover,

it also shows

good performance

in

processing strongly

structured patterns

interpreted

_as different

objects residing

_{on a} uniform

background

_as in

figure

3.

iiiiiiiiii _lllll'llll ~iiiiiii _{lll~lll lllllll} ⁱⁱⁱⁱⁱⁱ _{llllll lllll'}

lllll~liil _iiiiiii::I iiiiiii lJ~~ ff ^lll~ll

:::::::iii

i~~ ~ ~

8888j°8iI :~~~fl ^:::::~8fl ~~~fl ^f%:::ffl

(~~j%ill( (I(li:. _{((((1ii8~ H::::::::} ill((((("

fl~::::::: ::::::::~ :::::~flg :::::::::: ff:::@

Fig. 3. Samples the network is to memorize.

Figure

4 shows the recall

performance,

measured

by

the fraction of

recognized bits,

_{as a}

function of the fraction of

correctly presented

bits. It is not

surprising

that the

fully

connected

Hopfield

^network

(open squares)

is not able to stabilize this

strongly

correlated

training

set, since the differences in the patterns cannot be well

separated by

the

simple learning

rule and

N°5 ADAPTIVE ARCHITECTURES FOR HEBBIAN NETWORK MODELS 623

interference effects _are too

strong.

Note that

only

^six out of the _ten

samples

_are fixed

points

^of

the

dynamics

and hence _can be recalled

perfectly. However,

the

"intelligently"

diluted network

(full squares),

where 40$l of the connections have been deleted

according

to _our strategy, can

still recall the information almost

completely

_{even if up} to 20$l of the

pixels

have been

damaged.

~

_~'~

t

.

n D ° ^{° ~}

~ 9

~

~

~f m

lk- D

~ ~

~ fi

n-

~

~ ₇

o

~ .

j~

6

nc

~j

~ ~

5 .6 7 .8 .9 1-o

CORRECT INPUT FRACTION

Fig. 4. Recall performance of _a fully connected network

(open squares)

and _an optimally diluted network

(full squares).

7. Conclusion and outlook.

We have shown that the

performance

of

selectively

diluted Hebbian network models _{as mea-} sured

by

the minimal

stability

_parameter is

superior

to that of the

popular fully connecied

version.

Moreover,

the information is better

stabilized, although

one economizes _on the _num- ber of

couplings.

In

addition,

our

fully

deterministic

learning strategy

also finds

application

for

binary

and ternary

coupling

coefficients and could also be

applied

to

high-order

networks

[11].

^{Since the}

connectivity

structure of _our model system ^is

highly adapted

to the structure of

the information the network has

learned,

_one

might

try to extract

regularities

and correlations from the network

connectivity

_a5 well _a5 from the

couplings

which have evolved

during

the

learning

session.

Acknowledgements.

The author

acknowledges

support ^{from the}

H6chstleistungsrechenzentrum

at KFA Jfilich and from the German Science Foundation under contract number Se

251/32-1.

This work benefitted from

helpful

discussions with J.W.

Clark,

R. Folk~ H. Mfihlenbein and D. Staulfer.

References

II]

Hopfield J-J-, Proc. Nat. Acad. Sci. 79

(1982)

2554-2558.

[2] van Hemmen J.L., Phys. Rev. A 36

(1987)

1959-1962.

[3] ^{K6hler H-M- and Widmaier} D., J. Phys. A 24

(1991)

L495-502.

[4] Kfirten K-E-, J. Phys. France 51

(1990)

1585-1594.

IS] Ktirten K-E-, Parallel Processing in Neural Systems and Computers, R.Eckmiller, G.Hartmann and G. Hauske Eds

(World

Scientific,

1990)

pp.191-194.

[6] ^Bouten M., Komeda A, and Semeels R., J. Phys. A 23

(1990)

2605-2612.

[7] ^Derrida B., Gardner E, and Zippelius A., Essrophys. Lett. 4

(1987)

167-173.

[8] Morgenstem I., Lecture Notes in Physics12

(Springer-Verlag,

1986 399-427.

[9] ^Roucairol C., INRIA report 962,

(1986).

[10] ^{Fontanari J.F, and Theumann} W.K., J. Phys. France 51

(1990)

3375-388.

[1ii

Klirten K-E-, Lecture Notes in Physics 368

(Springer-Verlag,

1990 461-466, Statistical Mechanics of Neural Networks, Luis Garrido Ed., XI Sitges Conference _on Neural Networks.