Temporal Coding in Realistic Neural Networks

(1)

HAL Id: jpa-00247142

https://hal.archives-ouvertes.fr/jpa-00247142

Submitted on 1 Jan 1995

HAL is a multi-disciplinary open access archive for the deposit and dissemination of sci- entific research documents, whether they are pub- lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Temporal Coding in Realistic Neural Networks

S. Gerasyuta, D. Ivanov

To cite this version:

S. Gerasyuta, D. Ivanov. Temporal Coding in Realistic Neural Networks. Journal de Physique I, EDP

Sciences, 1995, 5 (10), pp.1367-1374. �10.1051/jp1:1995203�. �jpa-00247142�

(2)

Classification Physics Abstracts

87.10+e 87.22Jb

Temporal Coding in Realistic Neural Networks

S-M-

Gerasyuta

and D.V. Ivanov

Department of Theoretical Physics, St. Petersburg State University 198904, St. Petersburg, Russia

(Received

18 April 1995, accepted 20 June

1995)

Abstract. The modification of realistic neural network model have been proposed. The

model diifers from the Hopfield mortel because of the _two characteristic contributions to synaptic eliicacious: the short-time contribution which is determined by the chemical reactions in the synapses and the long-time contribution corresponding to the structural changes of synaptic

contacts. The approximation solution of the realistic neural network model equations is obtained.

This solution allows _us to calculate the postsynaptic potential _as function of input. Using the approximate solution of realistic neural network model equations the behaviour of postsynaptic potential of realistic neural network _as function of time for the diiferent temporal _sequences of

stimuli is described. The various outputs _are obtained for the different temporal _sequences of the given stimuli. These properties of the temporal coding _can be exploited _as _a recognition

element capable of being selectively tuned to diiferent inputs.

1. Introduction

Recently

_some models for

temporal

association in networks of formai _neurons bave been _pro-

posed [1-5]. Investigation

into the behaviour of these neural _nets have revealed

important

information

concerning

how _an ensemble of

interacting

^elements can

cooperatively

solve _com- putation associated with memonzation of recall of information [6].

Realistic neural networks _are

dynamical

systems with

complex lime-dependent

characteristics. One _source of

temporal

variation for these nets emerges from

dynamical properties

of

membrane

potentials.

Another _source _arises from

dynamical properties

of

synaptic

connections between _neurons. In

addition, synaptic dynamics

manifest two different

physiological

_processes

one associated with

depression

and the other associated with potentiation which _can _oc-

cur at different time

scales, resulting

in the formation of _a

complex

synaptic

strength

lime

history

[7].

W~j(t)

denote the synaptic eilicacious for the information transport ^from j to

neurons. It is _at the synapses where information _is

stored,

in that each

synaptic eilicacy

is

changed

in _a way that

depends

_upon _a _piece of correlation

information, namely,

_on whether the

presynaptic

_neuron bas contributed _to

firing

tl~e

postsynaptic

_neuron _or not.

In _our _paper the realistic neural network model has been

developed.

This model differs

from the

Hopfield

model

[ii

because of tl~e two cl~aractenstic contributions to tl~e synaptic

(3)

1368 JOURNAL DE PHYSIQUE I N°10

eilicacious

W~j(t):

the short-lime contribution which is determined

by

chemical reactions in the symapses and the

long-lime

contribution

corresponding

to the structural

changes

of the

synaptic

connections.

Short-time alterations of intemeural connections

W~j(t)

_are described

by

the two different

presynaptic

_processes:

depression

_a decrease of connection

eiliciency

after

spiking

of the

presynaptic

_neuron, and potentiation an increase of

eiliciency

after

spiking

of the

presynaptic

neuron. Each of the above mentioned _processes

decays

with _a different time constant, ^thus

providing

a

complex dynamics

of connection

eiliciency.

Tl~e realistic neural network

equations

are found in tl~e framework of

dynamical simplified

neural network.

Using

tl~e

approximate

solution of tl~ese

equations

_we bave described the bel~aviour of the membrane

potential

_as the function of time for the different

temporal

_sequences of the stimuli. The outputs for the various

temporal

_sequences of the

given

stimuli _are obtained. The present

study

allows _us to suggest that tl~e _appearence of

temporal coding

is determined

by

the short-time alterations of

synaptic

eilicacious.

In Section 2 is

given

the approximate solution of the realistic neural network model equations.

Section 3 is devoted to the calculations of the membrane

potentials

and the

synaptic

eilicacious in this model. In the conclusion, the status of the considered model is discussed.

2.

Approximate

Solution of Realistic Neural Network Model

Equations

In _our paper a

practical

_treatment of realistic neural network model has been

developed.

Each

neuron is connected to many other _neurons of the network _via

synaptic

connections, which _are

characterized

by

their

synaptic

eilicacious

W~j(t).

The

presynaptic

neuron j has contributed to

firing

the

postsynaptic

neuron1 _or not. This model takes into account two characteristic contributions of the

synaptic weight W~j(t).

The

synaptic weight

_can be considered _as _a function of two different _processes:

synaptic potentiation

Xi

(t)

and

synaptic depression X2(t).

Our approximation ^{of the} synaptic

weight

is defined _as follows:

W<lt)

=

lJ<o

+

J<lt))lXijlt)

₊

X2jlt)

+

Cj) Ii)

Here

Cj

represents the constant part of the

synaptic strength,

_J~jo is

analogous

to the

Hop-

field matrix [11. Tl~e

synaptic weigl~t

also indudes tl~e

dynamical synaptic

eilicacious

J~j(t)

wl~ich

correspond

to tl~e structural

changes

of the

synaptic

contacts. Tl~e

synaptic activity

is

Nj(t)

=

8(Pj(t) hj),

where

8(x)

= 1, x > 0,

8(x)

= 0, _x < 0.

hj

is the threshold of membrane

potential Pj(t).

The

synaptic activity

is

equal

to one if the _sum _over

W~j(t)Nj(t)

is positive, and equal to _zero otherwise.

Xij (t)

and

X2j(t)

_are defined _as follows:

xi~jt

+

i) xi~jt)

₌

-Aixi~jt)

+

BIN~ jt) j2)

x~~jt

+

i) x~~jt)

_m

-A~x~~jt) B~Njjt), j3)

Xij(0)

₌

Xijo, X2j(0)

=

X2jo

here1, j

_are labels of the _neurons and _range between and _n.

Nj(t)

is the _neuron variable and is 1 for

firing,

0 for

non-firing

_neurons.

Xijo

and

X2jo

are the initial

synaptic potentiation

and

depression correspondingly.

Equation

(2)

describes tl~e potentiation _process and equation

(3)

describes tl~e

depression

_process.

Every spike

of _a

presynaptic

_neuron increases

(potentiates)

the

synaptic eiliciency by

_some amount Bi This

potentiation

_can be

interpreted

_as a short-

lasting pl~enomenon

witl~ _a time constant

1/Ai Simultaneously

witl~

potentiation,

tl~e

spike

of the

presynaptic

_neuron

produces

_a ^decrease of the

synaptic eiliciency by

_some amount 82.

Thus the

phenomenon

is called

synaptic depression. Similarly,

the

depression

_is _a

short-lasting

(4)

phenomenon

with _a different time constant

1/A2. Together

these two processes can

produce

several types ^of ^rather

complex

behaviour under

appropriate parametrization.

The

synaptic weight

^also ^indudes ^the

dynamical synaptic

eilicacious

J~j(t)

which _are determined

by

the time constant

1/A. Jjj(t)

_are defined _as follows

Jujt

+

i) Jj~jt)

=

-AJ~jt)

₊

BN~jt)Njjt), j4)

where B is the model parameter.

The difference form of the

equations

^for the membrane

potential cil)

_can be considered _as

p~jt

₊

i) p~jt)

₌

-ap~jt)

+

f w~ jt)N~11) j5)

-pN~jt)

+

s~jt)

Here t denotes time, a and

fl

_are the characteristics of

dynamical properties

of menlbrane

potentials. S~(t) corresponds

to trie externat stimulus.

The

approximate

solution of the

suggested equations using

the

analytical

method is calculated. We _can propose the _recurrence for the solution of the difference

equations.

For

simplicity

this method is

explained

for the

following

_equation

xjk

+

lj

₌

axjkj

+

yikj, _16j

where

x(k)

and

y(k)

_are

arbitrary functions,

_cx is _a parameter, k

corresponds

to the time. If the substitution

x(k

₊

ii

₌

cx~+~z(k

+

1)

is

used,

that _we obtain

zlk+i)

=

zlk)+a~~~~vlk)

zlk)

₌

zlk i)

+

a~~vlk i) Ii)

zli)

=

z1°)

₊

a~~vlo)

Summarizing equations ii)

_we have

zlk

+ 1) ⁼

z1°)

+

O~~~~vlm)

₁₈₎

The function

x(k

+ 1) in trie

following

form is constructed

k

x(k

+

1)

₌

cx~+~x(0)

+

£ _cx~~~y(m) ₍₉₎

m=0

Using

the

proposed

method the dioEerence form of

equations (2)-(5)

for the realistic neural networks in the form

(10)-(14)

_are obtained:

Pj(k

₊

1)

₌

il cx)~+~P~(0)

+

j~ _Il _cx)~~~

x

(10)

m=o

x

If

_WV

_lm)Nj _lm) _flN~lm)

+

ilm)j

J=1

(5)

WV

lm)

=

lJvo

+

Jv lm))lXij lm)

₊

X2jlm)

₊

Cj) Iii)

m-i

Xij(m)

=

Il Ai)~Xij(0)

₊

£ Il

Ai

)~~~~~BiNj(1) (12)

1=o

x~~jm)

=

ji-A~)mx~jjo)+f~ji-A~)m-i-iB~N~ji) j13)

imo m-i

J~j

(m)

=

Il A)~J~j(0)

₊

£ _Il A)~~~~~BN~(1)Nj il) (14)

1=o

By

_an

analogous

method the

equations

for the

synaptic

activities

N~(k +1)

_are obtained

Nj(k

₊ ₁₎

= 8

iii

cx)~+~

l~(0)

hi +

j~ _Il _cx)~~~ ₍₁₅₎

m=o

x

É

WV

lm)Nj lm) flN~lm)

₊

S~lm))

Here the

synaptic

eilicacious

W~j(m)

^define the kernel of this

equation. Equation (10)

is

simplified by

help of the 8-function projection properties

(here Nj (m)

=

8(Pj (m) -hj )). Using

the smoothness of

positive

functions W~j

(k)

and the restricted functions

Nj (k)

_we _can calculate the

multiple

_sums in

equation (10).

Each _neuron _can be described

by

its _mean

firing

jate ^{fj (the}

new parameters ^which

correspond

to the

long

time average activities

f

=

£ _{Nj (m)).}

k + 1

_~

These parameters are not

dependent

_on the time and therefore

they

_can be

takÎ1out

_of

the

sum

symbols.

Then the

multiple

_sums _are calculated. At first

by

this method the

synaptic

eilicacious

W~j(k

+ 1) _are obtained

w~jk

₊

i)

m

iii Ai)k+i xi~jo)

₊

ii A~)k+i x~~jo)

+

c~+ j16) +lli Ii Ai)~)( ^Ii ^Ii A2)~)()fz)x

~IJZJO ⁺

( _Ii _Ii A)~~~)ÀfJl'

The

dependence Wjj(k +1)

of the initial chemical

strengths Xij(0)

and

X2j(0) disappears

at k _- _cc. For the

asymptotic

limit the

synaptic

eilicacious W~j ^differ ^from ^the

Hopfield

matrix

Jjjo

^because ^{of the} two contributions: _one _is determined

by

chemical reactions in the synapses and the other corresponds to the structural

changes

of synaptic contacts,

Wv

₌

lJvo

⁺

(Lfj) _(Cj

⁺

() ))

fj)

,

(ii)

where

fi

is the _mean

firing

rate of the _neuron.

Using

the

synaptic

eilicacious W~j

(m)

_we _can obtain the membrane

potential P~(k +1)

_now:

P~ik

+

i)

=

ii a)~+~P~io)

₊

Lixijio)inJvo ^li

₊

( _in _nA)f~)+ _i18)

(6)

+X2j(0)(Y2J~jo

+

~(Y2

^Y~~)

^f~)

⁺

^Cj(YoJ~jo

⁺

^(fi

^Y~~)

^f~)+

+()(Yo ^h) )(fi _Y2))Jjjo+

1 2

+

Ii

(Yo Yo~ ⁺ Yi~

Yi) (~

^(Yo ^Yo~ ⁺

^(~ ^{Y2)) fi} ^)fj

1 2

k

-fl fiYo

+

~j _Il OE)~~~S~(fil)

m=o

~?

^~~~

i/li/ll~-il~-i~~~~

^~~~~

~

^~~

~Î

~~

^~~~~~ ^~~~~

~

^~~

^~~~ Î

^~~~~~

~~~~

fi "

~~

°~~~~

(22)

cx

p=1,2

In the solution of

(18)

_we set

J~j(0)

₌ 0.

Using

the solution

(18)

_we _can _see

obviously

that neural network remembers its

"history",

1-e- the membrane

potentials

_are ^defined

by

the inputs

Sj(k)

for the whole time intervals.

3. Calculation Results

Using

the approximate ^solution ^{of the} ^realistic ^neural ^network equations we have descnbed the

dynamical

behaviour of the membrane

potentials

and the synaptic eilicacious for the different

temporal

_sequences of the st1nluli. The various outputs for the different

tenlporal

_sequences of the

given

stimuli _are obtained. We confined _our

analysis

to the dass of

fully

interconnected

nets with elements that _were

homogeneous

and

isotropic [8,9].

The membrane

potential

in the _case of the short-time alterations of

synaptic

eilicacious _can be considered _as:

P~(k

+

ii

=

P~(0)(1

_cx)~+~ ₊

II (Xi ⁽⁰⁾ ^~

^Yi ⁺

^X2(0)

⁺

))

_Y2+

1 2

+

(C

⁺

) )) _Yo)~n _flG)f

+

£ _Ii a)~~~slm), ₁₂₃₎

1 ~

~

~ # OE

^Î ~ _Jdo

Z>J"i

If the

long-time

alterations of

synaptic

eilicacious _are also

mduded,

then the nlembrane _po- tential

P~(k +1)

has the

following

form:

P~lk

+ 1) _"

P~lk +1)

₊

Xi1°) )) _In _Yfl)+

124)

(7)

lfK)

_Fig.1

6.o 5.o 4.0

~

3.0

/

Z-o i.o o-o

K

P(K)

6.o 5.o 4.0

~ '

3.0 2.o o D.D

K

S(K)

4.0 3.0

Z-o ^'

I.Ù ^'

Ù-Ù

0 5 10 15 20 25 30 K

Fig. 1. The temporal coding properties of realistic neural networks.

a)

The menlbrane potential mcluding only trie short-time alterations of synaptic eliicacious

P~(k

+ 1),

b)

trie membrane potential

with trie long-time alterations of synaptic eliicacious

P~(k +1),

c) The mput stimuli

S(k +1).

The parameters descnbing the membrane potential _are _as follows: _a ₌ 0.1, p ₌ 0.1, Ai ₌ 0.01, A2

= 0.02, Bi = 0.02, 82 _" o.01, ^C ₌ o.02, _~n ₌ o-1, _n ₌ 50, f

= o.2.

+

lx21°)

⁺

il

iY~ Y~~) ⁺

lC

⁺

t il _in _Yo~))nf~

Analogously

trie

synaptic

eilicacious

W~(k +1)

and

W~(k +1)

_are obtained:

W~(k +1)

=

iii Ai )~~~Xi(0)

₊

Il A2)~~~X2(0)

+ C ₊

(25)

+

((1 ^Il Ai)~~~)

(l Il _A2)~) (~) ^f)~

i 2

W~(k +1)

=

~

W~(k

₊ ₁₎

(~

⁺

~

(l ^il _A)~+~) _f~) ⁽²⁶⁾

~ A

In trie _case _m

question

_we believed that there _are _sortie short-time alterations of

synaptic eilicacious,

which _are determined

by

the chemical reactions in the _synapses and the

long-t1nle

alterations of

synaptic

eilicacious _are defined

by

the structural

changes

of the

synaptic

contacts.

The calculation shows that the neural network

forgets

most of the information obtained for the time k

mJ 40 since the moment when the input

disappears.

We take in consideration three

time constants

1/cx 1/Ap

I

IA

₌ 10 100. Our choice is based _on the

expenmental

(8)

Î'fÎ()

_Fig.Z

6.0 5.o 4.0

r

~

3.o Z-Ô I.o Ô.Ô

K

Î~ÎK)

o

K

S(K)

4.o 3.o

r

Z-o I.o O.o

o 5 la 15 20 25 30 K

Fig. 2. Trie model parameters correspond to Figure 1, but the _mean firing rate is f ₌ 1

data I?i ^and ^allows us to consider trie correlation between trie short-time and the

long-t1nle

alterations for the realistic _neuron networks. When the

temporal coding

properties of realistic neural networks in _ouf

approach

_are

observed,

the functions

W~(k +1)

and

W~(k +1)

_are

decreased

monotonically

with the lime for the different parameters of trie model. The

velocity

of the

decreasing depends

_on the three paranleters

Ai, A2,

A.

The

dynamical

behaviour of the

P~(k +1)

and

P~(k +1)

_as the function of time for the different

temporal

_sequences of the stimuli is shown in

Figures

and 2. In the first _case the

following

model parameters _are chosen:

cx = o-1,

fl

=

o-1, Ai

₌

0.01,

A2 _"

0.02,

Bi "

0.02,

82 "

0.01,

C

=

0.02,

A =

0.lAi,

B ₌

0.lBi,

_~n ₌

o-1,

_n ₌ 50,

Xi

(0)

₌ 1,

X2(0)

= 1,

P~(0)

= 1,

f

= 0.2

In the other _case

(Fig.2)

_we _set

f=1.

In

Figure

1 the

dynamical

behaviour of the membrane

potentials P~(k +1)

and

P~(k +1)

_are

compared.

We _can _see that the various outputs are

determined

by

the different

tenlporal

_sequences

ofinputs (in

_our _case

only

two stimuli

presented

in the different order _are

considered).

As shows in the

Figure

1 trie

dynamical

behaviour ofthe

potentials P~(k

+

1)

^and that of

P~(k +1)

_are similar and defined

by

the converted stimulus

£

k

il cx)~~~S(m).

The parameters

Ap, Bp

and others _can determine

only

the deformations

m=0

in

shape.

(9)

In

Figure

² the differences between the short-time and the

long-time

alterations of

synaptic

efficacious _are shown. Trie

temporal coding

properties _are lost for the

long-time

_case. The

dynamical

behaviour of the membrane potential P~

(k+1)

_is determined

by

the

synaptic efficacy

parameters A and B, which _are

dependent

_on the structural

changes

of synaptic contacts for the _mean

firing

rate

f

= 1.

4. Conclusion

In _our

approach

the short-time alterations of interneural connections _are described

by

the two different

presynaptic

_processes:

depression

_a decrease of connection

eiliciency

after

spiking

of the

presynaptic

_neuron, and

potentiation

_an increase of

eiliciency

after

spiking

of the

presynaptic

_neuron. Each of these _processes

decays

with _a different time constant, thus

providing

_a

complex dynamics

of _connection

elliciency.

^The

synaptic weight

indudes also the

dynamical synaptic

eilicacious

Jjj(t)

which

correspond

to the structural

changes

of

synaptic

contacts. These synaptic eilicacious _are determined

by

the

large

time constant

1IA.

^The

present

study

allows _us to suggest ^that ^the appearence of

temporal coding

is determined

by

the short-time alterations of

synaptic

eilicacious

Wjj(t)

The behaviour of realistic neural networks is considered in the franlework of

approximate

solution of the set of

coupled

nonlinear difference form of

equations.

We have used

only

_some

correct

suppositions

and the

approx1nlate

solution

(18)

is

suiliciently

motivated. We _can

point

ont that the _our realistic neural network has the

properties

of the

temporal coding.

We believe that the

temporal coding

is maintained for the different realistic neural network models.

Acknowledgments

The authors would like to thank Yu. D.

Kropotov

and Yu. M. Pismak for useful discussions.

This work _was

supported by

the Comission of the

European

Commumties in the fratrie of EC-Russia Collaborations under the Contract ESPRIT P9282 ACTCS.

References

[1] Hopfield J. J. Froc. Natl. Acad. Sci. USA 79

(1982)

2554.

[2] ^Amit D.J., Gutfreund H. and Sompolinsky H., Phys.Reu. A 32

(1985)

1007.

[3] ^Buhmann ^J. ^and Schulten K., Europhys. Lett. 4

(1987)

1205.

[4] Sompolinsky H. and Kanter I., Phys. Reu. Lett. 57

(1986)

2861.

[SI Nakamura T. and Nishimori H., J. Phys. France 23

(1990)

4627.

[6] ^Freeman ^J-A- ^and Skapura D.M., Neural networks: Algorithms, applications and programming

technique, Addison-Wesley Publishing

Company (New

York,

1991).

[7] Kropotov Yu.D. and Pachomov S-V-, Sou. lfuman. Physiologylo

(1984)

813.

[8]

Engel

A.K. et ai., Trends in Neurosciencels

(1992)

218.

[9] Gutfreund H. and Mezard M., Phys. Reu. Lett. 61

(1988)

235.