PULSE AUTO

DESIGN AND ANALYSIS OF AUTO SCALING PULSED ANALOG

NEUR.AL CIRCUITS

By

@Dipa nkarBhattacha rya,B.Tech [Ho ns.]

Athesis

submitted to the School of GradueteStudies in partia l fulfillment oftherequire ments for

the degree of Master of Engineelins

Facultyof EngineeringandAppliedSciences Memorial Universit yofNewfoun dland St.John 's,Newfoundland , CanadaAl B3:<5

August,1991

1""1 01""''''

^{Nationa llibrary} BibliolMQuo nalionale duCanada

Theauthorhas granted ann-evocablenon·

exd usivelicenceallowingtheNationalUbrary of Canadato reproduce,joen,<flSbibuleOf'sen copies of hislherthesisby any meansandin anyformorfonnat, makingthisthesis available to Interestedpersons.

The author retainsownershipofthecopyright in hislherthesis.Neitherthe thesisnor substantial extracts fromitmaybeprintedor otherwi se reproducedwithouthlslherper- mission.

L'suteura ecccrdeune~cenceirrCvocableat non exclusivepermettant aIaBib'iolh~ue natcne ieduCanada dereoodulre.pr~ler . distribuerouvendre des copies de saIMse dequetcuemanlereel scus quelqueforme quece soil poormettredes exemp lairesde cotta ut eso

a

la dispositio ndespcrsonncs inleressees.

L'auleur conservelapropnetedudroitd'autcur quiprotegesathese.Nitathese01des exlr.:lits substantialsde ceue-crnedoivcnt~l(e lmprbnes0tJecnemen t reproduitssans son autorisation.

Canada

Abstract

Minimizationofsynaptic areais important inaneu ralnetworkwilliahigh synapse to neur on ra t io. Conse quentlyone has tooptimizethe synepsc ra t herth' L1 1 thc neuron.Apulsedanalognetworkwithamplit udetnodulationres ultsill;\very compactandefficient synapse. Charge summ a tio nis usedwhichleadsto,I.single busasthe summer.Membranecapacitance hasbeen distr ibute dtothesyn iLpM's allowingthe net workto be perfectly scale d . Likethebiclogicalueuron.llll'neuron fires asingleoutp utpulse when the acti vation exceedsthethreshold. ;\ diseh;lrgl' pulse is generated todischargethe membr aneca pachanccsviallisc:lwrgctmusis - torswhichhavealsobeendistr ibuted to synapses [orscalingpurposes.Cirnlil designanddet ailedanalysis hasbeenincludedalong withsimulationresults.Stan.

derd cells havealsobeen presented.Asthe pro posedarchitectu rebehavosquiu- differen tlyfromexistingarchitectures,sim ulatio nofsomeof thc stnudn rdexam- Illesofneural net works have been included. Two chill" IHLvealsohL'('1Il!l'"igllt'd using3Jlmdesignrules.

Acknowledgement

Isincerelyacknowle dgeand thank my supervisorPro f. Br uce . Lock ha rt for all hishelp,useful discussions.criticisms and encouragement[orIlly work forthe wholedurat ioll ofmy programhere. I thankThe Schoolof GraduateStudiesof Memorial Universi t y,theFaculty of Engineer ing,the AssociateDean of Engineer- ing(Grad uate Studies)andhisoffice for thenecessaryfinancia lsup port whichI needed badly.Ialso thank all thestairmembersof C·CAEandspecially to Lloyd Liul cfor making my thesismorepresentable. atleastintermsof the figures .I

;..[so than kall my fellow graduatestudentsin the Facul ty of Engineeri ng.Fina lly.

Ideeplyacknowledge the constantencouragementof mywifeandour families.

Co nte nts

Abstract

Ack n owled gemen t

Cont e nt s Listof Figures

ListofTabl es

ListofSy mbols

1 Introduction

2 Ne u ro nsan dNeuralNet work s 2.1 Introdu ction.... 2.2 Neuron. 2.3 NeuralNetwork..

2.3.1 GeneralReview . 2.3.2 Leemin g: 2.4 Concludingremarks.

3 Literatu re Review 3.1 Introduction.

iii

vii

II 1.')

16 Ifi

:1.2 Review.• . . • •.

3.2.1 Analog Implementation. 3.2.1 Di~italimplementation.. . :1.;1 COllciuJin!rcmar ks.

.. DesignPhil osop hy&tProp o sedArch itect ur e -1.1 Introduct ion.

,1.2 Dbj eerives .

1.2.1 Mot ivations 1.:1 Proposed Architecture .1.-1 Co ncludi ngremarks.

5 Circu itDesign and Ana lysis

5.1 Introduction .

.i.:! Exc itato rySynapse.. . . .j.2.1 CircuitDes cription.. .).2.2 Ci rcni l~ ig n.

5.2.3 Circuit Ana lysis.. 5.3 InhihitorySynapse .. .. .

.5.3.1 Circ uitDescript io n .. 5.3.2 CircuitDesi gn. 5.3.3 Circ uitAnalysis . 5..1St a ndardNeuron

s.t.t Circu it Dc"cripLion 5..1.2 Circu itDesign_ . 5.·1.3 Circu itAnalysis.. 5.5 Input~CllrOn"...

2.

26

27 28 32

33 :l3 3J J3

3.

H

., .,

4. 4 .

46

·IS 46 51 53

~.5.1 CircuitDescript ion.

~.5.2 Design .\.:Analysis <.Ifstandard input lieuron . 5.5.3 Dl!Si!';n!:Analysist1{Inn-fling11IIJIIl X'-lIrtlll.

.

i.6 Concludinr; rema rks .

6 St andardCells

6.1 Introd uction.. 6.2 Cell Specificat ions . . 6.3 CellDescription.

6.3.1 ExcitatorySynapse. 6.3.2 lnhibitcrySynapse 6.3.3 Standar dNeuron. 6.:1.·1 Invertinginput neuron. 6.3.'; Stand ardinpn t ueurcn. 6..l Simulalion. • ... .

7 Simulat ionsandResults 7.1 PAttern Classifier. 7.2 XQR Gate. 7.3 Cccperetive Assignments. jA Implementa tion. . 7.5 CondudingRemarks

8 Con d usio ns

Referen ce s

Append ix

62

li:1

fiti Iii li~

ilj j~J

I/J(i 11I7 107

113

121

List of Figures

2.1 Structure of1l.claaslcelneuron (ada ptedhom(~lcad.S91J.

2.2 Blockdiagram ofa typical artificialnCIITOIi.

2.:1 Alypicalmulti-layer ed fccdforwardnetwor k.

4.1 lllockdiagra mofthe prcposcdneural archit ecture.

5.1 Schematicoftheexcitat ory synapse. 5.2 Normalizedfiringrateof theneuron withoutleakage.

IJ

:!9

34

5.:1 Aclh 'at ion cu rvesfordifferentvalues of r [whenlea kageis included),39 .",." Simulation res ult ofa sinr;lcsynapse

i".j Activation\'ollage generilled using equations.j.ll .5.16 and

s.t

j . H 5.6 Schemat ic diagram ofthe inhibitory synapse... . I.S 5.T Spicesimulationofthreecxcila loQ'andoneinhibitory 5)'napscs.. -Ii 5.$ SclH~malicof the standan.l neuron.. .

,

').9 Spicesimulationofthestandar d neuron•. . . 5.10Schematicdiagramorthe staudardinpu tneuron.

S.llSchema ticdiagramor theinverting inputneuron.

5.1:!Hspiccsimulat ionor thest a nd ardinputneuro n.

5.13llspicesimulationortheinvert ing input neuron.

6.1 Layoutortheexcitatorysynapse.

6.:! Layoutortheinhibitor)'syna pse.

vii

49 50 54 55

60 66

6.3 Layout ofthestanda rdneuron. WI

6..1Layout of thecomparator. in

6.5 Layou t of t hc buffe r 71

6.6 Layo ut of theinvertc ea.. 7'2

6.7 Layout of theinver te r:!.. i:I

6.8 Layou t cfthetwo input :,,\:\ \) gate. 71

6.9 Layout of t hcinver ter. iii

6.10Layout of ra m pgenera to r iuNO. i i

6.11 Layout of theinverting input1\"lIT'l 1l. is

6.1:!Layout ofillNI. SlJ

6.1:3Layou t ofthest andard iuputneuron. ,..q

6.14Simulationoftheextrec tvdluyou tof111t~,'xrit'UlIry~yn i' I'M'. :-\:\

6.15Simu lati ononrhoextra cted scheru a tir

or

tllt~standardn" lIn"'.. .~I 7.1 Sche m a ticofthete mplate flIiltl'hing,~.~illllplt " ,~ li 7.2 Spicesimul ati on of thete mplnteIlLlltchin g;exmuplc.

" 'i

7.:l Spicesimul at ion ofthetemplatemal ching example. .~.'i 7..1 Spice simul at ionofthetemplate ruat ehiugexam ple. uu 7...) Spicesimu lat ion of the tem p late matching('Xalll l,le. !Il

7.6 Content add ressa bleme mor y. !I:\

7.7 Outputof7 neurons when presentedwithIIIIOU.• !II

7.8 Schemat ic

o r

the firstXOI{circui t. !Ij

7.9 Plot sfor inp u t00. !J~

r.re

Plot s forinpu t01. ~'!J

7.11Schem atic of thesecon dXORcircuit. . lUll

7.12 Plot sfor input01. 101

7.13 P10ls fur inputII. IU'.!

viii

7.14Schematicuf the thirdXORcircuit.

7.1.) Plotsforinput 01.

7.16PloLSforinputII.

7.17JxJ cooperative.u~i&nmclLtnetwo rk.

103 101 105 108 7.18OutpntofAll9neuron.ofthe 3x3rooperat jveau ig r' incnlnetwork.109 7.19 Schematicofcontr ollablepatternc1Msificr;CAM.

7.20 Ilspiee simu lat ion ofthoCA:-'1.

i10 111 7.:!1I.ayou t of the networ kobtninc d by auto plnreandroute rontines. 112

ix

List of Tables

5.\ Controlvolt ages andthe periodsorthegenerat edpulsesfor 'he etenda rd inpu tneuro n .

5.2 Controlvolt.ges and the p nodsoftile&cllcrate dpulst..'Sfur the

invertinginputneuron. 5~

•.1 Weight distribu tion of the3:<3cocpcrntivc assignmentncr. 106

List of Symbols

Symbol

('(.'f)

('./

('.ISIV

r

Descrip tio n

·tran sconduc tanc eparameter - charge coupleddevice

·zerobiasjunction haltom ca pacita nce density ofthe moal bulkdiffusion -ze robias sidewallcapacitancedensity

- Farada y consta nt - bulkthreshold paramet er

II '; . .

rransconductnnceparameter forP~'IOS 1\':, -transconductance par ameterforNMOS 1\"' -t r a nscond uct ance parameter

·lengrholtra nsistor

- channel lengthmodulation para met er .H.I -hulkjunct io n bottom grading coefficient ,\/JSW -bulkjunctionsidewall gradingcoefficient M NOS •meralnlt rld e oxidescuuconductor

- surfacepotcmlal l:JH •built inpo tentia l /l .lI11i\'l'rsalgasconsta nt

\:" - membranevoltage

\; - threshold voltageof MOS transistor

xi

Vn .zero bias thresholdvoltage of :\10 Stra nsist or

":,,, • weightvoltage

tV,} • weight from neuro njtcneuro ui

W .width ofatransisto r

S

= !f .

sh ape facl or -valency

xii

Chapter 1 Introduction

Research concerningneura l networ kscanbetrecedhack1\few decades.butit ha~beenmostly onthe theoreticalstudies and computersimulation.Com p uter slrnulationisslow and the real powerofthe neuralnetworkcallhcstbe extracted ollly whenone geesfo rspecialized circuitsinmicroelectronics.Inthelast ten Y('iH~orso,"lo tof lite rAturehasbeen publishedon neuralcircui try and their implementa tioninsilicon (chapter3).To date.however.most ofthenet works reportedare smallinscale. The-cvm pu li ngpeweeofneural networks liesin their connect ions. In biological syste ms.one neuronmaybeconnec ted to tho usan d sof otherneurons.So.onehastoconside r the implicationsofthe scaleifoneisever toapproach the sizeofnet works presentillour biologiCilI.y.tem.

Oneneuronisconnect edto"notherneuron lhrllugha synepse .Ifthere ;Ire Nneuronsin ;I network,thenthe numberof synapsesgrowsas/lilfor a fully connectednetwo rk(such1\5a.Hopfield net).fromacircuitpoinlofview. it is net allthateasy toconnectalarge numberofsynapses togetherandfeedtheoutput toilneuron.However,for an aul o scalingcircuit, the numberofsynapsesper neuronisnotlimited byallYcircuitconstra int,

1\11aulaKalillgpulledneuralnetwork is presentedinthi. thesis.Itleads toaverycompactsynap sewhichishighly desirableinanetwork wheresynapses

out numb er neur o ns,Itfurtherenables cue toaddtill.'outputsof<Il~,rgeIIIU1\I,('Tof synapsestogether.The autoscaling feat urehasalso cuabled us todesignstandard cellswhich can bepluggedtogetherto realizenet wor ksof\'aryingsizes.

Pulse-stream ana log networ kshave already beenrepor tedill theliter a tllTl' (chapter 3).Und e r thisscheme,the neura lstate is representedLyPUb l' Swhose frequenc ydepen dsontheinputacti vatio n .Buttheproposed circu itsdiffl'l'in manyrespects.Thescalabilityhas beenachieved by distr lbn rlng the liIemhrane rapac it a ncesin the excitat orysy na pses. Synapses<:<111be either cxeit..tory IIrin- hibitory butcan not switc h hack and forthbetweenexcitationandinhib ition.Ear h time theactivationvoltagegoes past the vhrcshold,unoutputpulseis generat ed.

At the sa metim eone dischargepulse isalsu gener ate,lto ,lischargethel1It'mbr;u w capaci tan ces(like repclarization in biologica l neuro ns]sothat thedliLl"l!,!'in1.t'g ra.

tion cycle startsall overagain.For the scalingpurpose,dischilTgt,'r ;llI ~i sl. orsarc al~odistributedinsyn apses.

These neuralcircuit shavebee ndesigned, anti sirnulatcll lIsingtheSI,ict,pro- gram. De tailedmat hemat icalanal ysishas alsobL'C1Ldone.A1I1llHI,e r

or

stand a rd networ kslike pattern classifier,contentaddressable memo ry. XOItgiLt....~,llop- fieldne tsetc.have beensimulate d toexamine prope roperatio n oftill'desigllcd circuits.A sta nda rdcelllibraryhas also bee n developed.

Thethesis has beenorganis edasfollows.Thesecondchapter int roduces the biologica lneuro ns followed by artificialuouralnet works.TWIIblTliingSdll: lI1t·S havealsobeen included the re. Thenext ch apterdealswit h thelitera turerevlnw. howdiffere nt aspectsoCne ural ne tworkshave beenachieved by differentpeople.

Thefourthcha pte rgivesthe design philosop hy.Thc proposedneural architecture is also presentedthere.Thefift hchapt e rgivesthecircuit desig niU lllanalysis.

Merhe m a ucal equationsarcalsopresentedwhichcan be used for developing aCast

simulator.Chapter six deals with the standardcellsinclud ingthecell design phi- losophyalongwith layout ofsever&!standa rdcells.The seventhchapte rdescribes slmularlonresults of diffe rent neuralnetworks.halso inclu desthesche mati cdi- agrll.mAnd layoutof oneof the two chips thatwouldbefabricated. Fina lly,in thelast chapterIconclude mypresentworkandSllUestsomeareaswhere further workcan be done.

Chapter 2

Neurons and N eural Networks

2.1 Int r o duc ti on

This chapterdeals with11 brieflntroducrion tobiologicalneuronsandarlilicial neura l netwoks.A brief discussionallthe neuronfollowedbya simple descrlprion on the generation ofactio n pot entiali~present ed. Dil£cfI'1I1~~~PL'Ctsoftn-ura]

networksincludingtwopopularlearning schemeshavealsobeenincluded,

2.2 Neu ro n

Theneuronis thebasicAnatomica1 unit of the nervous system.:\ tYI,inll,,,,ural cell(figure 2.1 )hasfourdistinct regions - cellbody,t1clIclrih..,..axonandlIlt!

presyn apti cterminalsofthe axon.Thecellbody isthe source ofI'lll"r~'rut11Jt' neuralinform ation processing.ItgivesTis" 10 "tubular11m•.:...".,kuowuit.'Ih.!

axon whichca.nextendoveritlargedistance .TileaXOIl, illturn••livi.k'S intoiL

largenumberofpresyna pt icterminals.Thes e prcsyneptlctermillabCtm tad witll the pos tsy n apticterm ina ls(dendrit.:~)of till;othe r neuro nsattheSyll;ll' licsill'~.

Theneuron int e grates the incom ingsignalsIecmother conncctlngucurcnsby lhe capaci tanceof thecellbodyAnd fires en out put pulse(actionpotential)whenthe tolal inputactivation exceedssomethresholdvc.hllg....SOllieilXOIiSarc':QVl· n ..1

withaninsulating materi alcalled myelin to reduce the capacitance between the cytoplasmand theextracellu larfluid.Thisis essentialfor achievinghigh speed couducticn.Themyelinsheathisinterrup ted at reglilarintervalsby the nodes of Ranvierwherethetransmittedsignals arcperiodica llyrestored.

Nervecell,likeothercells,has different conccurrationsfordifferentIonsacross itsmemb rane (Koester,st].Qntof theions,Na+andct-concentrationsare lowerinside whereas1\+andorganicA-arelowerouts ide.Due 10 theconce ntre- tiongrndlent,1(+ions tend tomoveout acrossthe membranethrough diffusion.

Thi.sdiffusionleads toseparatio nofcharges anti hence a potential difference

n·:.. )

whichimpedesfurther passage of charge.At a voltageof around

·is

mY,A'+

ionsreachan equilibriu mwhen thereis no net now of1\+ions. This equilibrium potentialcan beobtainedby the Nernst equation:

('l.I)

where

C:

and

ct

arethe concent ratio nsofionsinthe extracellu larfluidand insidethecell.

Dueto the presenceof NaTions. the cellcomes 10 aresting potential of about -60 mV whenthe netinflux ofNa+ionsis totallybalanced by the neteffluxof 1\'+ions. Inorde rto maintain the ionicgradient.a metabolica llydrivenNa-K pumpbrings in asteadysupplyor1\' " ions while driving Na+ions out of the cell.

Ifthe membranepotentia lis increasedfrom ·60mV 10 say -iOmv.thecell is hypcrpcla rlsed reducing its ability to generatean actionpotentialand is therefore

~<lidtobe inhibited;whereas, ifthe potential is decreased.the cellis dcpolariscd

andis saidtobe excitedbecause it increasesits abilitytc generatethe action potential.

Ifa nerve cell isdepolarisedto a smallextent . the cha rgeleaksaway and the actionpolentia lis neverinitia ted.However,ifthe cell isdepolarised to approxi-

">,

=-~~~!;...-=--=:; .

...''"....0'"

~."" '_ ~ I"

..._

)

Figu re 2.1:StructureorItclll~~icalllcurun(lId"plcdfrumt~'lcad.ti91J.

mately-40 mV, an actionpotentia lisgeneratedevenif thevolla ge isbrought ba.ck to·60 mY.Thisgenera.tionof actionpotl"T1tia l ca nbe explained in termsof the v" h&!e dependention chan nels [Koester.BIA).When tilecellisdepclerised.Sa'"

iUliehannelsopenlincrea.singNa+conductance )thereb yincreasing inwa rdSlI + current .This furtherdcpolarizes the e:t:1I ....hich in ruruopensmoreNa+channels.

This regenera tive processec nrinuesliII the ecucn potential;5generated.How- ever,al this stage,Na'"conductanceand henceNa+currentsta rlsdecreasing rl'!iull ingin furth erdecrement of,vll+conductalice.Al the sametime.1\"+ion r1Hlnnds open resultingillan outward/\.+currentwhicheventually repclaeieethe metnbrene to thetesling pote ntial.

2.3 Neu ral Network

2.3.1 Gener alReview

Artilicialncu ra lnetwerksarebiologicallyinspired.They l\t<.'net...orksof simple prcc•.'ssing elcmcru s crunilsinterconnectedby weigllu of variahleSlrell SlllS.They arc neur al in thesensethl\t the computa tionisdone colil'Cti\"Cly rather than individu ally.In gene ral, in a neuralnet work, an amplifier withanon-linear output characteristicform s the cellbody,wires replaceaxensand dendrites.andthe resisters modelthesynapticconnections orweights amongthe interactingunits. When aneuronis activated.itevaluatesallinputsfromotherneuro ns andfinds outtheweighted sum.Ifthe sumortheacrlvaricngoes beyondapredeterm ined threshold. thene uro nchanges its output (figur e 2.2).

Inmath ema ti cal terms.irOJrepresents thesetorallne uraloutput s.then the lotalactivationnet;ortheit hneuronis

(2.2)

O i+-0~

02~;2

^/

^'-,

-,

....-... ~

-",

" ":"' " I "-- , -,;

0 3--7---<:~ L / _ ./ /

^r,

~ ^/ /'----'" - : t; .n Iu nc l ion

04 ~\/ ^{/ odd er} oc t ivo t e ,

, / On +-<3/

Blockdiagralll of alypirlllll.rlilicil\lll t·UIOIl.

Figure2.2:

where W,!isthe strengthof connectionfrom unitu_Jto unit11,_The outputofII, is givenby0;=f(acl;, lhr}wherefis a nonlinea ror decisionmaking func tion iLlltlt hris thethreshold voltage.

The strengthofconnection between two neurons determ inesthe degree of interaction between thetwo.IteMIbeeitherexcitato-y orinhibitory,norma lly represented bypositiveand negativeweights.If theconnection isexcitatory,then theact ivationoftheneuronisincreased, while theinhibitory connect ion tends to reducethe activation.

Whenanetwork is activated,all the neurons operateinparalleland try to adjusttheirstat es. In thesynchronous update procedu re.theysimultaneously update theirstales ateachpulse of a centralliming clock;whilein asynchronous update,eachofrhoneurons. at anyinstantoftime. has1\fixedprobabilityof updating its state.Sincetheneuronsupdatetheirslatesindepende ntly.in a very smalltimeframconlyonc neuroncan be thought of updating itsstate. Whatever tlll~updat ing procedure,eventu ally theneurons settletoa stable steterepresent- ing someglobalconfiguration.Thisis achieved byutilizin g thelocallyavailable information andthemassive parallelism inherentto the system.

Oilferent researcher shave proposednetworks employing differentunitsindif- Icrcntconfigurations[Aertser.al.,891but1I10stcan be encompassedwithinthe staledframework. The majordifferencesare noted below.

•Conuectivity: Connectivityvariesfrom single layerednetwork[e.g.Hop- field nets)to mult ilayerednetworkswithhidden units (e.g. backp ropnets).

Bnckprcpag atlonnets arcalso strictly fccdforward and theco nnections are essentia lly unidirect ional.lIopfieldnel s,OILtheotherhand,have bidirec- tionalconuoctions. Bot harc discussedin more detail below.

10

•Neuralunits:Networksemploying sim plelinea r units haveaset'Ifinput units andII.set of outputunits.Itca n beshownthatthe rtllllpul ati ulldUIIl' bymultilayeredline a runitscan alsohedone bya network with ou t11Ilid ,l"ll layer. The outputofthe simple linear IHolielis allidentityfuucuou:th.uis OJ""act;.Koho ncn has donecxtcus ivo sunlicsonthiskiud nfnotwnrks;11111 theirlearning [Aarts d.;11., 891. 011 the utherhand.illllll!lhu-arthres hold unit,output0;

=

Ilftheacti vat ionudj>0;{whe re0,islht·l h rt'~IIUI.1 1;11111

o

otherwise.Percept ronsarc/Ispl·cial dass ofnetwo r ks,:ml'l"~'illgil.~i llgl, · layerline a r thresholdunit swithoutanyf.>.!.lhack.Burthe most':01111110111

one is the one utilizingthescmilincaractiva tion lunc-t.iouwheretho"litput O.

=

!(rul, j,!being Rrnnllotollic"I1~·lInll.d,·creasi lig,litf" n' llli alJ[,!Iunct.ion.

•States:Outp utst a t escalleitherbehinary:i.e..0, :(U. IIillwllidlClls,'till' funct ion!ismaking aharddecls iou asintIl<'purceptruumudd.11uplidd\

contentaddressa ble memory.backprupilgilti o lLnetworks;OIl'I,ll!'ulItPllt ..an

lng funct ionasinHopficld'sucnral decisiu ll ncLwor ks.

•Activation:TheoutputIuuerlon orthed.-'cisilJnfiliinlsu1)('"ith" r,h'IN- ministicorprobabilistic. Themodelsc1l1pluyi1L~tlwfonner;H<:ll"l'lid<l nets,backpropagati onnetsetc.Whcrt!;ISth,!Bo)UW lill lnl;wllirll:mllplo)'sa probabilis ticresponseIuncr iou.

• Represen tation:TheoverallrepresentatiullI;JlI\lit'local.illwhid ltill:st;ll,~

ofindividualunits mayrepresentsomethingmeaningful.011the contrary, in thedistributed re present ation,thest at e ofeachunit11Mtoheilltl!tpr etl:d inconj uncti on with all other neurons .

Itisworthwhile,in th iscont ext,todiscuss1I0plidiinets. Ill"lloplid,1net.

ever-y neuronisconnectedtoeveryot he r neu ron exceptforitself[l.e.IV,i

=

0).

The otherrestriction is thatthe weightsare symmetric al,that isW;I=:Wj ; .F'or

;1two-stateneuroni,the totaliuput is

(~.31

whereI;is externa linputto the lieuroniand~is theoutput of neuronj. In thesimplest,non-grndc d form u la tion,the outp utofneuroniisVi

= v?

ifx,> V, andI·;Qotherwise;whe reV,is thethres hold forthe neuroni,An energyfunction suchas

E=

-~

^L:

L:T;Jt~\~

^-L:I,V;

⁺ L:U,I~

^(2.4)

,t-J ' ,

limy he associated with the netwo rk[llcpfleld,82],The n thechangein the energy.

:lE,duetothe change inthe output of ncuroniis

- I L:T"V, +

I,-

U ,I"

V;

-I',

-Ud" V; ^(2.5)

The ahov(!quantity is alwaysnegativebecause ifX,>V"then~v;is positive:

othe rwise both ofthemarenegat ive.Thus anychange inV.lowe rstheenergy func t ion. SinceEisbounde d ,the systemeventuallyrea chesa stable state when nomoreoutputschan ge. Asimila rexpressionfortheene rgy funct ioncan alsobe obtulnudfor neurons wit hgrade dresponse[Hopfield,841.

2.3.2 Learning

Till'informationcont e nt in a neuralnetworkresides in theconn ec t ion strengt h.

Learning is the proces s of edjusungthe connection st rcngr hsor the weightsin such away as to prod ucease t ordesiredoutputs. Lea rningcan bebroad lyclas- sifiedinto supervised and unsu pervisedlearning. [IIsupervised learn ing , inputs

11

arepresent ed along with a setofteaching inpu ts. Weights arcalljllst(~1s1q)by stepunder thesupervision of theteachinginputs so11l'1.t the network will pro- dnce acorrectout put pattern when everthetrai nedinput panem isprcsr-nu-d, Inunsupervisedlearnin g, there is no teaching input.However,tllt~11;' 1work learns by capturingthe regularit iesoftheinput patternsand res ponding to nuy s[ll't'iill featurethat mnybe present in the input pattern s. t\ briefreview of thelxu-k propagati onlearningscheme(superv isedlearning)andcompetitiveICMlling (lin.

supe rvisedlear ning)follows next. Adetaileddiscussion011these twoI'~;lrnillg schemescanbe found elsewhere[Rumelhart ct.al..8.\1.

Backprop agation neura lnetworksarc strictlyhierarchica lrl'('dfurll"an llHulti- layered networks(figure 2.3). Thefirst layer is Lhe input1;1.~erwhichrl:n' il'l's exte rn al inputsandfeed s the outpu ts to the next layer orhidden units.Anylayl'r canreceive inputsfrom the layerjust before it and can project theOlltPllts tn the layerimmediatelyafterit.There may hemorethan onehid-lenlayl:r nnd uur- outputlayer. Hidden and output units,employingscmlliucar act.ivat.ioumil'S.are useful for capturing higherorder regularities.Besidesthese units, there may also be bias unitswhicharc alwa.ys on ami areconnectedtothe hiddenandoutput units.

Backpropa gat icnlearning involvestwo phases of computation. It basically mini mizes (gradientdesce nt)the sumsquarederror over all theoutputunits and allthe training pat tern s. Inpu ts arc presentedami thenetworkcomputesthe outputs(O"j). Theseout puts arcthen comparedto the desired or the teaching inputs(I"j)togeneratethe errorsignal61'J wherethesuffixl'represents any patte rn p andjis any unit. Weightsarc then a,ljustcdfor allthcCO!lllcr;tiuns feedin g theoutput layer according to

("2.1;)

Input toyer Hidden toyers Out outtoye r

Figure2.3:A typicalmulti-layeredfeedforward network.

wheretIisthe lea rningrateandOpjis theinput to the unitjfromtile unitifor the patternp.Itcan beshowntll11.t forthe 011tPUtunits

wheref'is the derivativeof the activationfunction.6'sarethen computer]fur thepenulti matelaye r accordingto

(:uq

where m'saretheunits connected totheunitk.Thuslhe erro ris prup ;L!;"t",1 backone layer.By utilizingtherecursive Ionuulacfequatiuu(:.!.S),the crrnrrun be computedforany unitsillala}'l'randthewI'i~h tsarc a(\justcIIilcturll ing1(1 theequation(2.6) . Itisimpo r ta nt to1I0 te 1hat thepatternsMe required tobe presented repeatedly inorderto generatethe proper internalruprescntut.iun.

Thetypical activation functionemployedbythehiddenMIdtllC~UUlI"l l,units is give n by

(:U l j

Thisis asigmoidfunctionwhich isdiffere ntiableaswell.Tile,l" ri v"ti v,~off,l' is givenby

~«~;, ⁼

^{O,{l- 0;) (:.!.!U)}

Thisderivativeis maximumfo rOJ=0.5andsincethecha nge ill weightdcpeuds onthis derivative,weightchangewillhI! maximumfor tll"Llnil.~witl!outputsncar rho mld-rangc.

Inthecompetitive learning method, unitsMeabo organised illilhicrard,iud layeredfashion. Any unitscan receiveinputslromallthe units;11rhelayer immediately belowand canfeed the outputtoallthe unitsillthe next upper layer,throughexcitatoryconnec tions only. Allthe unitsinala yer ..rugrouped

I'

togetherinto anumber of cluste rs.Units in a duster inhibitea ch other so that only oneunit in a cluster is active(~winncr·tllkc·all"strategy). Each unit has a fixed amountof weightdist ri bu ted over allthe inputliliesi.e. [jWi;

=

I.

Learning is achievedbyshifting weightsfrom the inactiveinput lines to theactive ones.IfII.unit wins, then each ofthe inputlinesgives up a proportion(1\ ) ofits weight which isredi st ribut ed amongthe active lines. That is

toW;} 0 ifunitiloses

I\'!; -

^{!{Wj )} ifunitiwins (2. 11) whereL_J=Iifthe input line from unitjis active and n is the total number of the activeun it s.Howeve r,ifthein put pat ternshave fell"active components, then some of thelinesmay never be on and the corresponding unit may never win. In order to removeth a tconst raint , the weightca n also be changedaccord ingtothe aboveequationeven if the unit loses,but at a much lower proportion.This at least,willenablethe unitto bein thecompetition.Itcanalso beachievedby c.mnglngthe thresholdin such a waythaLthe unitbecomesmore sensitive when it loses andbe comes less sensitive ot he rwise.

2.4 Concluding remarks

In thischa pter,blologlcalncuronsand neural networks have been discussedvery briefly.Simp lemodelsofthe artificial neural networkshave beenpresentedalong with two learningschemes to introducebasic ideas about neuralnetworksand the relevantterms thatwould be used throughout therest orthe thesis.WitlJ this done. the next chapteris devotedto aliter at ure review.

Chapter 3

Literature Review

3.1 Introduc ti on

Rese archconcerningneural networksca llbetracedbac k asFa ra.~the ]!l·lOs.

Since then (exceptfor a briefperiodinthe end ofthe tiD'sandtilebcginulugof the 70's) a lot ofworkhasbeendone011neural networksbutmostlyinvol ving theo ret icalstudies andcomp utersimula tion.Simnlnrlonofla f !;!'ueuraluetwork is very slow-mostly because of largeCOllllcetivilit.'silIlWIl!\thefOllllt~'liligdtmll'uts and sequentialcalculatio nandupdatingof neural states.T11I~actualprnl11isl'IIf neur alnetwo r ks,however, is in specialized hardware.especiallyin minudec1.ro nic circu it s. Theil one ca llpossibl yexploit thespeedandpowerorneuralnetwork and goforpracti calapplications.Themajor obstaclesin realizing uenrulnctworks insiliconwere thelack ofavailabletechnologytil do so alit]suHicil~ntkllowl(~,lg"

on struct uresandbehaviorofneurons in nervoussyste m.However.a greatdeal ofwork has alreadybeendoneonthelWTI'OllSsystem!lnd th.. mln' lItilndrap id progressof very large scaleinteg ration[visi)systems hasmade it possiblethese daystoreal ize large neural networksinsilicon. Anumberorresearc hers are workingonthedesign and implementationorueuralnctwcrks and alargeHumher of papers has alreadybeenpublish. ad.

16

Re viewing ofthisliter aturecan be done in a numberof ways.One way is to tackleeach of the designissuesseparatelyanddo a comparativestudyon different appro aches. Alte rnat ively, the designscanbegroupedtoget her onthebasis of tl,('technologies [e.g,digital, analog, mixedanalog-digital etc.] and a study doneof eachgroup.Sincethe proposedcircuit s arc analog, stress will mostly be Oildifferentimplement atio n issuesofanalogcircuit s includi ngmixedoranalog.

digitalappro ach. Some of the majorproblemsofpuredigital design and some cleve r solutions of theseproblem s willalso be presented.

3. 2 R e vi e w

3.2.1 Ana log implem en t a t io n

Inananalog circuit,thesum ofthewciglltcd produ ct canbeim plement ed ina very compac tarea.This particular aspecthasat t ractedmany designersto go to analog circuits .Nevert heless. analog circuits suITerfrom variousproblems. First of all noise immunityand immu nity to processvariabilityis very poor. The other not.able drawbackisitscom para rlvclylow precision.Thelatter one ispart i<; ula rly problematic for variouslearn ingschemes which needwe ight adjustmentin very smallsteps. Multiplica tio nis often achieved bytheresistor s, whichsuffer from several drawbacks. Curre ntsummat ionis usua llyemployedwhich can suffer from saturatio n problem.Ontopof that.someoftheanalogcircu itste ndto bebulky.

Analog neur alnet works have beenquite throughly discussed in[G rafct.al.,891.

Themajordesign issues one shouldconsldurfor the impleme nta tionofanalog neura lnetworks are :

I)fixedV5 .program mab le conne ctiv ity :!)realizat ion of coupling strengt hs 3) volatilityof connectionstrengths

IS

4) type ofconne ctio ns 5) sizeofthe neur al co m ponents 6) easeof fabrica tion.

7) learning

Fixedvs,program mab leco uue c t.iv ity:The COllll'IlLiligpowe-r

or

allC UT;, 1 networkdepend sonits connectivitywhich inturn depends ontheprohlemrho networkis meantto solve. That is why most of theimple!lll.'lIliLlionsan'appli- cation spe cific. The circuits designedby Graf cr.ul.[Grefct.,II..S7,S:']!law program mable connect ionpauerns.The neur ..1lIU1.l11ll, insl.l'il.11offl~'dingsnlllf' otherneurondirectly,controlstwo switches. TIll) connection is ('ompl l,tt',lthrouglr two otherswitches which are controlledbythecontent of two ram cells.TIll','IJlI' tent oftheramdeterminesthe type of connection - itca n he millIeexl'ilatllry, inhibitory or leftunconnect edcorresponding to a content of+1, -I orU.Thustlu- connectivityof the netw ork can be chnngcd by changiug themnll'lll uf thl.'mills and hence, differen tconfigura tio nscan be mopped intothe S,II11enetwork.

Coupling strengths :Reali zat ion of the coupling strcugrh is an importaut issue because itdetermines the network's abilit yto learn. ~I{)sluf thel'"r1il'f designs[GraJct.al.,87,881,[El-Leithyd.ul.,871usedfixedvalue resistor-,as the coupling elements. Althoughthis islile simplestwayto realize networks, thcre are afewdisadv antages to the app roach.First, differentconnectionstn~ngl l l.~IIl'l:,1 differe ntvaluesofresist o rs and he nce, differenL silicon areas.Thisprevent.1Ilf' networkfromhaving1\regularstructure aswillnormallyIll' achie\'I!dwith fixed size co uplingelem ents(synapses). Then, once fnbncatcd.therl'sistorscannothe altered anymore,freezi ng the state ofthesystemso thatkaruingcannot lake place no r canthesystem be reprogramme d.Since thepa tt erns tohl~stored are

19

oflennotknownapriori.thefixedvalueresistorapproachdocs not coveraswide arange ofapplicat ions asonewould normallyexpectfrom aneuralnetwork. Not onlytha t,resistorsareexpensiveillterms of siliconarea, particularl ythe high value resistorsreq uired to keep the overallpower consumption ofthe circu it lo w.

Grafct.a].(Grarct.aI., 871 ha vede veloped aprocess by whichamorphous siliconca n bedeposited (asresistive elements )onanotherwisefinishedchip.Vlsi compat ible highvalueresistorsusing thin filmhave alsobeenreport ed [Hubbard ct.al.,861.Theseresistors,packed in a chip,callbe used ro replace theresistor mat rix in a network;however,the size oftheresistor packis severlylimitedby the pin count of thechip .iftheprecise valueof the resistoris netimpo rtant, diode connectedtra nsisto rscan he used[El- Leit hy ct.al., 871.

Varia b lecoupling strengthhasbeen achie ved in[El-Leithyet. al., 871by adjust ing the th res holdvoltage\.'Jof theinp uttransistor.\.'Jdependson anu mb er of parameters,most ofwhich areprocess depende nt(e.g.gate matt-ria l.gate insulationmaterialandit 's thickness ,channel do ping etc.].Italso de pe nds onthe bulk(su bs t rate )to source potential 1158of thetransistorin a non-linearfashion.

By chang ingVS H •~~and hen ce thecoupling stren gt h can becha nge d.Howe ve r, this requi resa variabledebiasforeach ofthe connections andis difficulttorea lize even foramodest numberof neuron s .It canbe generatedon chip, butauto m a tic control willreq u irearather compl excontrollingsche me.

A circuithas beenimp lement ed usingi\INOS/CC Dprincip les[Sage et . al., 86} achievingthevariable couplin g strengthin averyelegantway. Thecircuit works uu twuconcepts-chargecoup leddevicecont rolsthe movemen tof the ch a rge trans - nuttedby a syn ap seandthe i\INOSdevicestore sthesynaptic weight ingvalue.

The chargepacket releasedbythesynapseis mo du latedby the trappedcharge underthe~INOSgate and a mete redquant ityis availableat the ne ura l outp ut

'10

forgenerating the activation.Bythe application of the external'"0 11age.variable amounts ofcharge can be storedin the nitridelayer of the~I N OSstructurethus achievingdifferentcoupling strengthsor weights.

Variable coupling strength call also be achieved((~Illrra}'ct. ;\1.,~~)lJ,hy dynamicallyrtoringthe chargeon a capacitorrepresenting theweight\'UltlL.'!;t>, [Brownlowet. al.,901have used switchedcapncit or techniques to \"I'ali1.I'flilly programmableweights, Wei;l:hts arc stcre.l ill capacitors11m]Meswitchedhy transistorswithspeeds determinedby theincoming pubt' rates.Bilmlilt\\'\'iKht.~

havebeen realized in[SchwartzC1.<II.,S!JIby storingthe weightstlilr'~fI~l\tiillly"II a pair of capacitors. This scheme abo considers weightdecay.11111has .n-hicvcd10 bits of analog depth for the weights.

Program mable bistable switches/resistors based on dilrerentcrystalline mate- rials orBism ut h.oxidehave been reported[Spencer. 861"By alJplyingplll~cs,llll' conduct ivityof the mater ialcan be increasedby severalorders or magnlurdc.It can bebroughtback to the initialinsulating suuc byapplyinglIegali\'(~pnlsos.

When electric field is applied,vacantoxygen sites arc rroatedwhichcUlil rilllllc to the conductivity.By suitablebiases and applying pub t's. rl'sist ivity "r the requiredvaluecan be obtainedandhence can be used as a progrnuuuableCOli- nectl onelementsfor the neuralnetworks.This scheme seetus to be anillll:rt'.~Li\l~

propositionbutrequires alot of improvementOilthe metallurgy or these mau-rials so thatil would be possibleto realize a large scale i!Traywilhidentici!.lswitrllillg characte ristics.

A two quadrantmultiplierwith a digital weighl scheme h...s been descrihed in {Hollisct.al.,901.Weightis represented 11)"

.1

set or parallel binaryw,·i~lll.,·d (W

IL

ratiovaries inbinary fashion] current sources.

Floating gatetechnologyseemsto hethe mostviablewdghL slorageSc1)l!UlC.

21

It hasbeen successfully used in the ETANNchip[Video ,91/ and gives 6-8 bits precision.Under thisschemeISl e,SI).chargeis injectedfro m thesiliconacross the firstofthetwoinsula tors and stored in thefloaLi nggategiving riseto athr eshold vuhage shifl. Progra m mabilityi~cllSilyachieved by storingdiffe rentamount of chargeinthe floa ringgat e. MNO Sis a similar dev ice buthas a differe ntstructure.

Vola til ityof theconnectionstrength: Nonvolatility oftheconnection strengthisimporta nt because once a propersetof weights is learned.itshould be retained forfut ureuse.Resistorsarcbest suited for thispurpose.Thisis

"Iso easilyachievedillSage's approach [Sagect. al.,861.Thecharge thatis tra ppe d underthe nitridelayer has avery highretent ivityatthe normalope rat ing conditio ns. The nOiltinggate approachorFAMOS isalsovery muchsuitable forlong term chargestorage,ln[Murreyct.nl.,891,tilechargestora ge being dynamic,thereisst eady leakage of chargefromthecapacitor.Periodicrestoration ofcharge is done [Brownlowct. al.,90]fromoffchip ram through a digitalto analog converter.

Typeofcon necti ons: 1\10stof the papersbeingdiscussed here use bothexci- tatory andinhibitory synapses. One common way ofrealizinginhibitorysynapses ([Graret.aI.,86, 88),[Tankct.aI.,86,8il)is to useth e inver ted outputof the neuro n.Inthepaper[Verleyeen et.al.,89],asimpledigit al controldrives all excitatoryCur re ntthroughone lineand all inhibitorycurrent through theother line depending011thesignofitcontrolline, lnhibitlon in[El-Lelthyet. aI.,8i]

is achievedby usingPMOS transistors. Inhibit io n is aboachieved in[Murrayet.

al.,8UIhyremoving charge from the capacitance,thevoltageacrosswhich repre- sentstheactivityof theneuron.However,onlyOIlCty pe , namelythe excitatory connc crlcn hasbeen achievedin [Sagect.al.,86J.

Sizeof thene u ralcom pone nt s;The areaof theneural compone ntshas

to be small in order to accommod atea large useful networkin achip. Sinn' the number of synapsesis usually much larger tha n tllMofneurons.OUI'll;\ ~to minimize thesizeof the synapse.The~INOS/ CCDcircuitis very compar t;1l111 soarccircuits describedin [Brownlowct.al.,90].In[Cotl er d.11.1.,SS).fuw neuralbuildingblocks have beendesignedwhichcan he usedallt'lln tage()ll~lrtu realize neur a l networksinvlsi.Analog computerswitha.number of vlsil'hips inconjunct ion witha hostcom pulerhave heen discussedill[Ebe rhar dt d.;11., 89],[Muellerct.al.,89J.A numberof chi psca n becon nectedtogether to realize alargenetwork. Fu ncti onallyboththeschem esareqlLill~com pet ent hutthey requirecomp lex control and timingandCIl 1\acconnnodato only,tsmallnumber ofneuralcom ponents perchip.

Easeof fabri ca ti on:One lias toI,,~rardulabout(:!loosingthebasi,'n)l l i '

ponentsso thatthey can be fabr i(atedllSingthewith-ly al'llilahl,·fahri,:aliun processes.Thecircuitin [Sagect.nl.,So)employsspecia l Iabrlcaucu1,~dlllil[lI'~

forrea lizing the~JNOSdevice. Resistors1\["Crea lizedill[em fcr. al..Stil hyII specialfabricationtechn iqueand alsoin{Hubb ard et. al.,S6]altl,,)IIghilwas claimed tobeavlsicompatibleprocess.l3i,ilab leswitchesandresist ors[SpI·lln ·t, 86)also requirespecialfabricationproced ur es.

Lea rning : Since the workpresentedill this thesis docs notcons ider\c;lrllill:\, learningcapa b ilityofdifferentcircuitswilt uot he discussed.

Pulsed analogneuralcircuitsIallund crthe analog category11I1d arc0111'ofthe mostattractivecandidatesfor neuralnetworks.t\va riety of techniquessuch us pulse widthmodulatio n,pulseheight morlulatinn .simplegat ingelr:.C.1111)('llsed to multiplythe pulsestream by theweightvolt age . Pulsed circuitshaveh'~I~ll reviewed quitenicelyin{Murrayet.al.,!}I]ll ndpu lseheight fllollillalilinseems to bethe best ca nd id atefor thi spurpose. Underthlsschem e, anulog wcigllt"oltage

isstored on a capacitorand incoming pulse ismodulatedbythis weight voltage thr011gha~IOStransistorIMurra y ct.al.,891.

3.2 .2 Digital implem entation

A puredigit,11approachtothe implementa tion of.ieura lnetworks suffers from a fewdrawbackseventhoug hit hasquiteafew positivepoint s that makes itan attractive call1Jirla te forvlsl system .Registersarcneeded for storing the weights.

Digita lmultipliersanrladdersarc rC<luin..-dttl obtain thesumofthe weighted produc t. Allthesearc expe nsivein tenns of siliconarea. Time sharing is oneway (Ifta king ca reof thispro hlem hutthis calls fora complexcontro lschemeand at thesam elime reducescomput a tional speed. On theotherhand digita l circuits arcrobustwith respect tonoise and process variati ons.They arewell suited for application swhe reprecisionis more impo rt antthanthe complexity orthe size andarc part icularl y ve ry wellsuited for variouslear nin g sche mes.

TheJigital approachis problematicfor a fullyconnectednetworkand ismore suitedfor alayerednetwork. Thisis so beca useat each connection,oneneeds an adderand a multiplierandtheyare expensive.Howe ver.di[e rent app roachescan be ta kenlo overcome theseproble ms. The nextfewparagrap hs dealwith some (J(theinnovative approachesforrealizingthesumof theweig hted produc ts.

t\digita lnc uro-chlp with sixneurons and eight yfour syn ap ses hasbeen de- scriucdin[Hiraiet. al.,891.The neurons operat easynchronouslyandseveral chipscanbe connectedtogethertorealize net worksof anyarbit rary size. Synap- tic weightsare programma ble(64levels) andcan be setormonitore dby a host compuler.Theincom ing pulse density is transformedto adensityproport ional to theweightbytherat emultiplier.Anup-down coun ter isusedtorealize exci- tationandinhibiti on andaratemultiplie ris used togener a tetheoutp utpulses.

This scheme.even thou!hunique initscoeccptioa, is\'l.'rybulk~·.

Adi&ita.l pulsedensitymodul~tiuncircuithilsbeen11~'5i~n,-..1an,l,1~,,:;rrillt.J in ffomberll;d.al.,

001.

£«1.chipcanbe1lSl..JitSalllallll·ill.lne,It...-ier-orrail hecascaded to form a largernetwork. Instead ofUllin! uonnathitMry ;uil lullMk numbers, pulsedensityarithnlcticIwhereeachhitII~s~,Xitl"tl~'~"I....I ....-ill;hl)IIlIm · bcrshavebeenused resultine:in"sirlll,lecontrol alillaritllllldir.·n.i~ 11It.~\,...·11 achieved attheexpense of a t;reaternumberof hits thanisn"'luin·dinnorlll;ll binarj-arithmetic.~lult i plicati"l1 i~achit.'l'O'd by,l"illg x-cr ;11111;ul,litiullisI,il....

rialthlls the computauon limels dln-cr.vproportleunlto tIl<'1l1111l1...r"f,It'urt,nS [Icra fully connectednetwork),

The circuitgivenin[Bbyoct,ill..~~JlIiHSrealized i'Iullv «muo-n..11It'l\\~Jrk withs)"Slolic architect ure.For;.: neurons.:!~''''psHr"I't'(l"ift,!!III.·"tIIl"II,·lll~' sumoftheweightedptcduct,Thept'rrOrlllall(t.'rauhI'illll.rol\l'(ll,}"illtr..,llIrillr, pipeJinin! buttheapproachneeds\'crycomplexrircu itr~'11.11,1controls.

Amultilayeredneural ud litect urcIIsio.!: rcllulararra.vl !I.u II('('n~;n"llin (Faureet.il.!.•89J.Eacharn.yisccenectcdto it'sfour it(ljaCt"llln~-i&h1,..r.IIlln",&11 ci[!:hlbi-direcricnalbuffers.Each cdlconsistsufaroutinp;partililflilproce..~illr.

partand byloadin[!: appropriateffil"SS<I.!C). anycdlcallh.:lo~i(ally...11111.'(1 ...11.0, any olhetcell.

Inanother bitserialapproachIBlltler ,.ot. al.,.:!!JI. l';IChsYliaptirc1t'IlWlIla.I,ls itsshare orweightedproducttotheIlilrtial slimhuerunningdownthe synaptic column, The outp utstateisft'$triclt'(l lu .'j,lil fl'T' ~n tlevelsandthemuhipllcation bythewei! htisachievedhy shiftillg thebinaryweight.

Adifferentapproachhasbeentakenill(Wcillflc1d,I5!J1wherethenellralullt pul statesare stored in acircularshirt register\\"li ichc an hesi mullalIt'OUSlY IlCCl'S1icd hy allthe neurons.Asimultaneouspartialpotentialis thusobtainedateachsllirt

lJ('Cratio nofthe register (forII.fullyconnectednetwork). Out the whole neural circuitassuch is verybulky,itincludes an adder,comparator,sixty-four9 bit weightstorage areasde.

3.3 Concluding r emarks

Tllis chapterdealtwithdifferelLtcircuits andimplementat ion techniquesforthe Jl(:'tIralnetworks. Somecircuitshave certainadvantagesinsomeofthedesign i~'pectsbutdisadvantagesinot hers. Floatinggatetechnology seemstobethe most suitablecandidate forprogramma ble.longtermanalogweight storage.The synapticcircuithallto becompact comp aredto theneuroninorderto achieve highintegrab ility.Withthesein mind,thenextchapte rdeals wit hthedesign phi- 10000phyandthe motivat ion behindthedcslg u orthisIlarti cularkindorcircuitry.

Chapter 4

Design Philosophy & Proposed Architecture

4.1 Intro du ct ion

Inany neuralnetwo r k,wheth erartificia lorbiological,thenumber(IfSyn i1llSl>S is much higher than thenum ber ofneurons. In a fullyconnected ndworkof /Ineuro ns, the number ofsyn a pses grows asIl~. Whentilt'lmph-menuulon in siliconisIlLhand, one has toconsider theimplicatlou'Ifscaleveryrnrcfully.'1'110' number ofsynapsesper neuron should notbelimitedbyany circuitconstreuus.

The pulsedanalog neural circuitsbeing proposedhere. haverlistincLiuJVilll l.ag{'S overmostof the existi ngneura l circuits.

4.2 Ob j e ctives

The mainobject ivesof this designapp ro acharc

•La minimize thesynapticarea

•to developan efficientway ofadding alarge num be rofsynap ticout puts toget her

26

27

•tu design standa rdcellswhichcallbeputtogether torealizeany neural drcuit.s independent or synapseto neuronratio-i.e.scala bility.

4.2 .1 Motivations

~Iillimizlltionor thl.'.~ynnpticareais importantbecausethe synapses predomin ate in anyneura lcircuit. lIowevl.'t,it isproblemat ic to add theoutputa of8.large number of synapsestogethe rantithenfeedthesum to the neuron input. Con- vcurionaldigitalandanalog circuits sufferfrom various drawbacks and they have

;llrcadybeendiscussed illchapter:J.

Pulsed analogcircuitsseemto be the most effectiveway ofreali zing very oompnct andefflcicntsyuapses[Murrayct. al. 911.Underthis scheme, the neura l slate isrepresentedhyatrain ofdigitalpllbes the frequencyof which depends UIItill'inputactivation.The height oftheincomi ng pulsesis modulated by a lorall)'storedanalog weight voltage. Ifthewidthof the pulseisnarrow,the modulatedcu rrentcan be thoughtof as a charge packet and canbe dumped on IIIa capacitor.Ifalarge numberof synapses areconnected together,more and morechargewillbe dumped on thecapacitortherebyincreasing the membrane I'ullagesteadily, Inordertoovercomethesaturation,thecapacitancehas to be increased ill proportiontothe uumbcrofinputs.

Ano ther advantageof this approachisthattheinfonn nric n contentisillthe frequencyofthe neural outputpulse,notin its height.Sothe out putcan be routed to adiat nntsynapsevery easily.NULonlythat,the pulses,being essentially digital,can 1Jt" resto redbymeans of digitalbuffers while beingroutedover11large llislallf e.Thesamebulfcrcall also be used to handlethe lanout problem.

Theotherpoint worthment ioninghereisthat thesynaptic circuits a.re either r-xr-itatory or inhibitorybut not both.Thatis, theycannotmove back andforth

between excita ti onand inhibitionas theycallinmostofthe exbting neuralnet- workmodels.So far.thiskind ofsy n a psehas neverbeenobservedillbi()I\J~kal neurons[Pe reonnazct. al., 86l.This bipolarityCUlleasilybohandledilldigili11 circu itsbutis problematicin analogimplementations .Forinstnucc,wlll'rea11111)' tiplierisused forthe weight circuit.oneneeds a four'luau rantversionIllsl,I·,,,1or a singlequadrantifbipolarityis allowed.

Finally the transformation ofchargepacketsin the syuanso is ;\I·hie\·.,,1II)' ap plyingthepulses toaMOStransist orwhosegale is held at the \\"eight\·oIL"!;,'.

Sothe amoun tofthecharge beingdllmpc.1onthe cnpacuor.1,:pl'lllbUlltuc r;IL.· (an dwidth)orincoming pulses andtheweight voltage. However,sine"tln-:-'IQS transistoris inherentlynonlinear.scaling tileweight voltagewillno t s""I.:LIII' neuraloutput linearl y. So far,thereis no l'viden ce tha tliuearity islI)<liula illl,,1in thebiologicalsystem.Moreove r,all thelearning:mechanismsemploys"nl(~surt of feedbackwhere theweightis changedtillthecorrect output is ullLaiued.This does notdemandlinearityso longasmrmotonicityis preserved.Evenirilturns out that linearityistherule inbiology.itIll"!be worthwhiletilallowlIt)Il.1iII ' ~iLril.y in orderto achieve a very compactand efficient synapse.

4.3 Proposed Architecture

Figure 4.1 shows the basic architectu re of the proposedpulsedaM]o gneural 11"1.- work.Forthe reason describ e dlat er[chapt er 7), two ,Iifrcrcutryp csor11!~llr"IlS have been designed . Thefirst one isthesta nda rd neuronIISC']ill tIll']dlld.~u ]ity(~r andtheout.p utstage.It consistsof11compara tor MI{1apairofpulseg,'n,'ratms which emit onepulseeach,every timetheinputactivaviongoespast thet11n'.;h·

old.The othertypeisfor inputneuronswhich'Irerate gene rators.Duekim!fin's atamaximum rate withan inputvoltageof!)volts awl gradually tl"':re1\5"5 tl",

29

Figure '1.1:Block diagram

or

the proposedauto scalingpulsedneural network.

rateasthe input \'oltage goes down.\VhcrCil$.thc inputneuron with a.:ird~·at theinput(not shown, pleaserefer tofigure i.l) istheiuvcrung type which Iircs at the maximumrate when the input is aero and decreasestl1l' ril-Ie,15theiU]lllL voltage goes up. Thissecond type cl neurons existsin the retina whichfI'Sp'lII'!

to darknessinstead of the light and arc calleddarkcells.

The channel resistance ofthe synaptic rranai.ror is controlled hy tilt'w,~i~ht voltage,thus controllingthe amount of charge nowtoandfromtire Ilwmlm'lIe capacitance.Ifthe synapseis cxchatorv. dl1l.rscis added to the cupar-itor while it is removedfor theinhibitory synapse.ToachlevosGllability.th,~1111'1I111ra1le capacitance has been distributedovertheSYl11lPS'~S. So, thetotal"Illliu:itance of the neurondepends on the number ofsynapses altac1letltotheneuron.IftI...

capacitance were included inthe neuroninstead.the sizeof rue cupacitance would have to be changeddependingon the 1I11111bcr of SYllilPSCS in Imler to avoidtire problem of saturation.Anotheradvantage istha t each synapse adds or suhlTill:ts its own share of chargethus avoldlng the problemof currentdensitybuild-up,hll' to simultaneo usarrivalof pulses.Since all these capacitors are in l'iHilllcl,tl...

addition essentiallyturns into a single minimumsi~clint' or'l.us WIJrl'Sl'nlillgthe inputactivation or themembranevoltage .

The otherpoint to be notedhereis thatthe inhibitory synapSI'>t'[0 not ClJn1.i1ill anycapacitorsandthe scalingis appliedonly lu tilt,excitatorysynapses. Sinec tIre inhibito rysynapse removes charge from the wt,llmembrane capacitnnrctlwrehy makingit harder for the neuron to overcomethethrcshuld,thereean heno1I1~llrHIIS withinhibitory synapsesonly, Theyexis t only to inhibit tlw t'xdtlllioJlandthis is achievedby removing the charge, not byincreasingthe cnpnchnuce. Ho wever . the scali ngof the inhibitorysynapses is obtainedinthesecondary level int1r,~

sense thattheremustbe more Inhibitory synapsesas tile number ofl~xcila.lOI·Y

:Jl

syn a psesgoesup.

Once the neuronfires.the associatedsynapt iccapacito rsare discha rged50

that the chargeintegrationcycle can startonceagain.Scalabilityhas onceagaln OC~IIachievedbydistributing thedischargetrans istors intilesynapses.Though a larg e numberofIrllnllis lo narerequired, all of them ope rateinparallelallasingle wid etransist orwhosewidt his scaledupbythenumber of syna pses.Whereasif Asingletransistorisusedintheneuron,itswidthha.stobeadju sted according toOn!numberofsynapses-thus mltking itimposejbleto go for the standardcell approach.An extrasetofconnect ions fromthe neuron output tothe sy na pses are nee dedinordertobroa dcast the discharg e pulse.buttheyca n runin par allelto the wire!carryingthe synapticout puts togetherto theneur on.Thus. thechannel willbeslightlywider toacccmodate atwowirebusinsteadofone.

SincetheamOlll1tofcharge being dumped onthe capacitordepends onthe pulsewidt h,itisrequiredthat thcoutput pulsebenarrow.However,forproper dlseha rgeoperation,thedischar gepulsehastoI,esi8nificantlywider.Thatis why twopulsegene ratorshave beenincluded .

Sincetherewillbenneurons ands synapseswhere s>n, synapse5'1will receivea dischar gepulsefromthe neuroniandan outputpulse from theneuron j.Thus,onanaverage,output and discharge pulses aretobefed10almostequal num ber ofsynapses.Sincethisnumbercanbe verylarge,fanoutproblem shave tobehandled. Adigita l buffer has been designedh...vingtILesame heightasthe syna pse .Twobuffersoccupy roughlythesamearea.I\S 1\synapse. Consequently, the bufferscanbeinsertedin thesyna ptic ranksvery ceaily,andthe signa.lscan IJt.' routed throughthe butlers.

Theresisterin thesynapserepresents theleakageforthe properoperation oftheneuron.Without it.the neuronintegr a testheincomingpulsesindefinitely

;t !

reducingthefiring ratealongit.layerednetwork.B}' diilt ributing theleakagealong with the capacitor.thetimeconsta ntbasbeen madeindcpeudell\ orthe scale.

Althoug h some circuits forweight manipulation have beeu designed,1I0tIIIlLdl workhas beendone on that and conseq uentlyit willnotbeincludedinthetht'l'i~.

The floatinggilte technclcgySCCIJ\Stobe the mostpromising candidat e Icr Ihe implementa tionofthe efficientweightstorage(chapt er3)andtill'~~'I\;IIJSCIHi~

beendesignedwiththatinmind,

4.4 C oncluding remarks

Inthischapter.designphilosophy fortheauto scalingneural nrchtccturc hus been presented.Therehas been significant deviationilltheproposcd erchh ecturcIrcrn the existi ngones. Membr ane capacitance hasbeendistributellilltheSYll;lllSI~

for the purposeorscalabilit}'.Theneu ronfiresonlyoue puisI'when tltl'il1l' lIt IlCtivationexceedsthethreshold voltage. AdischargeIIul)(.' is abc gClleraktl tu discharge all the associatedsynapsessothat thec11i1rgeilltL~riltiollcyril'can stilTt once again, Scalabilityhas once againbeenachievedbydistrihulill(l;the disdl;H&I' tra nsis tors over thesynap~.Thus scabiliryhas been achievl.oJ atthe l'XP'!IISC"r a slight increasein thesynap tic area.Two different inpu tneurons havealso11t.'C1L proposed forinterfacingnetworkstothe exter nalillpill s.'1'111'l1O!Ur;11architecture havingbeenprcpoeed,thenextchapterdcalswitlltllcuL'Sigll1t11l11t1la'y~i~uf the individualblocks,

Chapt e r 5

Circuit Design and Analy si s

5.1 Intro d uct ion

Thischapter con tainsthedesignsfordifferentneur alcircuits, namelythe exci- tetorysynapse ,theinhibitory syna pse andthesta ndard and theinput neurons.

Simulat ionofthe circuitsusingSpice,anda mathcrnetic elanalysisof eachof themhas a.lsobeen provided.

5.2 Excitatory Synapse

5.2.1 Cir c u it Descrip ti on

The excitatory synaptic circui tis showninfigure 5.1.The circu it usestwo min- imumsizeNMOStransis torsinseries.Theexcit at ionvolt age\<~..isapplied to the 1:ateofthe firsttransistor MI and the secondtransistorM2is gated bythe weight volt ageV"".Thedrain of MIispulledhigh.Theoutput of thesynapseis the mem branevoltageV... takenacross themembran e capacitance

e....

Tra nsistor

~I:J(minimumsize again) isthedischarg e transisto rwhichdischarges

e",

when- ever theneuralinp utactivation(or themembra nevoltage]exceedsthethreshold voltage.Transist or M4.isa longtra nsis tor which generatestheleakagereq uired for theproper operationof the s)"napsc.

33

3,1

Figure5.1:Schemat ic ofthe excitatory synapse. All transistorshaw'W= 5.'1/1 and L=3,1 exce ptfor M4whichhasL=20,6 'I,

Ifonly onetransistor(Ml) wereused insteadof MlandM2,theexcit atory signal would have to beappliedtcthe drainofMi. Sinceaneuron may hi!

connectedtomany synapses, thiswould require alargedriving capabilityfrom theneuronoutpu t. Not onlythat,whentheinputsignalV...is offandV",is greater thanzero, the drainbecomesthe source.Thegatebeing pulledto the weight volt age.therewillbe asteadycurren t flowingfromthe capacitor

e",

to the ground.Sincethe gate currentis negligible,theabove approach takes careof bothproblems.

The amount ofcurrent flowing throughMland M2 dependson thegate voltage of M2and thevoltage across

e",.

By controllingthe gatevoltageV",I.the amount ofchargethat wouldbedumped onthecapacitorcan be controlled. So theeffectof the excit atorypulsefromneuronIlitoneuronu,.throughthesyna pse$iJdepends on V",r,V",r,therefore,isthe strengthofconnectionbetweenthetwoneurons.

TransistorMlis always insatura tionbecausethedrainis being pulledhigh, For mostol theusefulweight vollagerange(described later),M2willbeinthelin- ear region.However.thecha rgingcurrentis somewhatlessdueto higher threshold voltage becauseof non-zerobulkto sourcepotential. Thisis not aproblemfor tileproperoperationofthe proposed circuitbutcan be taken care ofby tyingthe substr ate to the source pote ntialolthetransistors MIandM2.Chargingcurrent can alsobeincreased byincreasi ngthe width of MlandM2.Butminimum sized translsto raarc good enoughfor thisapplicat ion.Minimum sizealsoreducesthe parasiticcapacita nces .

Chargeis dumpedonlyduringthetimeV...is high . WhenVOi'is low,~H is cutoff but asmallleakagecurrentflows through thereversedbiaseddiodes betweenthe source.the drainanJthesubstrat e.Thisleakage currentisvery small (::::: JOpA)and can be ignored.Thisis becausesynapses will operatein

parallelandthe amo unt ofincomingcha rge(=::5/11\lexcila lorrsrnapo<e)willl....• muchmore than the ehargc lostdue to unwanted leakage.

5.2.2 Circui tDesign

Ifthe leakag e isnot includedillthe synapse.the neuronollll"ll periOiI is

T=T.l +l~ (' .1)

whereT"andT~are discha rgeandc1large timc res pecti vely.Ifthen :ilre"~'x["iti\- lorysynapses,andifluis thelimeilvcragl·d chargear rivalrate,then till'Loud arrivalrate isnl,," ,Fornsy n ,l p ~e5 ,thetotalmembranecilpllcit<LIlft.· isflC~..Su the ill/eragecharge timeis

1~

⁼

IIC"'II~:. =

C...

t:

wheret~isthe thresholdvoltageof the neuron.The;Iwragefirillg ratc is {5.:0

15.:1)

Normaliz ingthe aver age firingrateto the maximum firiug ra teII /T.l)andtill' average ra te ofchargearrival to theaverag cmaximum arri\'altalcI.......cuegelS

R

I

1 /1~ =

^I

+

C••

6Q..,4.

l...u isthetheoretica l maximum rate audis glveuby

(.'i..I)

(.'i) i)

whereQ",oris themaximum ral eof chargetransferhyl\sirlglcsyna pse excited althera teofitt,and witha weightvoltage of.".1volt s.TIn:abovecqu a riou is plot ted in figure5,2andbasicallyrepresents the activ ationfunctio n.TIll: curve doesnot exhib itthe twodecision etetesnorma llypresentillneuralnet wor ks.TIle

0.9

~

^0.8

!

^0.70.6 E

~

^0.5

]

^0.'

J

^0.30.2 0.1

00 0.1 0.2 0.3 0.4 0.5 0.6 07 0.8

37

0.9 Normalizedchargearrival rate

Figure5.2: Norm alizedfiring rate oftheneuron withoutleakage.

:JS

implicationis that a minimalcharg'! arrlva lrate should be mahuained(in order to overcomethe leakage as inthe biologicalneuron]to generatean ')1Itp1l1.

However, if aresistanceR1kis added in parallelto the capacitance ,

(.i.li)

The steady state solution isV

=

1".R ,k,[flo.R1k<VI,the devicewillnc\·.~rfire.

Vcan be representedby the standa rdexponentialequ ation

(a .i}

The charging timeT.is the timetakenttl chargenpto I,; , tIll'thrt·~hol<l\,.,Itagt' , So,the output firing rate is

R

=

1j, _IllkC,..:II[1

0;;1

if1••R^1k>t ;

o

otherwis e ('l. S)

Normalizingthe equationonceaga in.OIWgCl~

II ,

ljT d

=

1- rlllll-

d::;1

^(.'i.!J)

whereT

=

~andI,

=

~

=

~.f,;Scurrentthresholdwhich deter.

mines the firing insta nce.Fig urc ,).3showsilser jes

or

plots for11''''0.:1andvanou, values of

e",

(thatis differentvaluesof r].Itcan he seenthatrinfluencestill:cur- vat ure ofthe plots. This activationfunction clearlyshowstwo dcdslonstates hut isnot sigmoid.Thistypeof activation functionhas been describedillIRulI ll~l h;,rl ct.al.,841.

The capacitor

e...,

apa rtfromthe bulk membrane capacitance,includes Lht' parasitic capacita ncesas well. These parasitic capacitancesste m Ircm lIlchulkto drain capacitance of transistorsM.J ant!~Handthebody to sourceca pacita llce

39

0.10 0.9

0.8

E

^0.70.6

[

0.5

~

^2.00

-e 0.4

t

^0.3

~ 0.2 0.1

00 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

Normalizedcharge arrivalrate

Figure5.3: Activation curvesfor differentvaluesofr (when leakageisincluded).

·'0

ofM2.These capacitances arc voltageand.geometry dependent .Theex p ressio n forthesecapacitors are givenby ([Geiger ct.al., 901l

C CJ. A CJS Vl, P

BD""[i (VFl¢ sl},\/ J

+

[I (1'r!'; Bl],\'J S\y 15. ' 0)

The minimum sizegeomet ry isL "":J/lmand W=5A /!m .W=SA/lmischosen to avoid thedog boneeffect at thedrainand the sourcecontactpoints. Thetotal parasiti ccapaci ta ncecomesto around50to 60fF.Sincethisisquitevariable (process aswellasoperatingpoint dependent) ,membrane capacit ance has heel!

chos entobe100 fFor O.lpF,brin gingthe Lola lmembran e capacit a nce Lo....roulle!

O.15pF. The incomingpulse widt h hasbeen set toli.anssotll/\litsluglcsynapse with aweig ht volta geof-5voltsca n maketheneur onlire one pulsehutnotifthe weightvoltage islessthan 3,6volts. Thethresholdvoltageof the neuronliasbeen chosen to beequal to1.5volts whichis half way hcLwccnthe upperand lower valuesof lowandhighlogiclevels.The usefulweight voltagei.~fromV,.I='L5 volts to 5 volts . The lowervel.te is dueto the fact that M2cond ucts(iguurillg subthreshold operation)when VIUIismorethantheneuron thresholdvoltage(=I,,'j volt s) plus itsownthreshold.

Themaximumchargethatcan he delivered to thecapacito rbyonesingle pulse has beensimulated tobe equalto ·137

rc.

T~was scLLa:lOnssothaLtile maximumcurren t that can be provided by a singlesynap seis14.6/111.IfR,~is chosento be 500 kl1 thenr=2..'jand It= O.'.!1. Thisisillgood agreement because 1"'4ris the upper boundof thecur rent.

Transistor1\.14replac es flu,and hasitconst ant gatevoltageof1.5 volts.~U will be in sat uration forany membrane voltage morethan1.5.0.7(threshold volt age] or 0.8 volt.However,whenthemembrane voltage islesstha n 0.8 110Il . M4 isin thelinear region and theleakage curre ntisless.To compensate for thisfluct uation ,IIisset 10%higherthan wasderived andit'sabsolute valueis

41

0.23*14.6pA

=

3.4pA.In saturation

I, =0..5.k'.54.(1.5-O.i) 2

ts.ru

wher e 5-1is

'f.-

and is the shapefactor.5,[turns out tobe equaito 0.26. With W=.')Ap.m,LcomesLa20.6 11m.

1\Spice simulationofa singll'synapse with 6.,5nsexcitatory pulsesand5 volts weightvoltageisshownin figure5.4.

5.2 .3 Circ uitAnalys is

Refe r ring to the figure5.1, for11.weightof5 volts,tran sistorMl will be inSAt- urationand ]\·12willbein the linearregion.Thecur rent throughMIis given hy

(5.12) andthc curren r thrc ug b M2 is

Ncgk'f"Lillgthe pi\rasilicca pacitances atthe junctionof Ml andt.12.onecan say 111;11 II

= /.,.

The th resho ldvoltage of 1\11is gillenby

whereV is the source voltageof~11.Similarly,thethreshold voltageofM2is

(5.15 )

Solv ing the equatio nfor/1

=

12and notingthatV,'s can be taken to be constant Forsmall timesteps, V callbe writtenas

\,"=

~(a

^+b)±

~ {("

^{+(,)2_2[a2+2V",b_V",}

2])~

^(.5.16)

~ ^JUUll

-IJULlJU

] ~ ^;

-.N,,;n ~ ,

.. . ,

klnnru

^{1 " n n " n n}

Figure5.'1:Simulat ionresult of a single synapsewitlii\IVci,l1,htvoJlilW'of!l volts and 6.5nsexcitatoryanddischargepulses.

43

wherea:::::v"r -\{landb=V",I-V11.This value ofVcanbe put backto theequa- rlcn5.13 for/1 tofind the inatant encouscurrent through~12. Ifthe timesteps /LTCtaken to be small enough,the n

(5.17)

Figure5.5showsthe out put ofa smallC-progra m(along wit h Spiceoutput for a comparison)tocomput e the output of a single synapseduetothe excitatory pulses (widthis 7.0n5 andperiod isaOns).lt agreesquite reasona blywith the sim ulationresultsfrom Spice. The sm alldeviarieuisduetothe factthat the program does notconsiderthehigherorder effectswhich arc present intheSpice level;] simulat ion.The other point tobe noted hereis thatthe leakagetransisto r hasbeenomit ted.

5.3 Inhibitory Synap se 5.3.1 Circ uit Descrip ti on

Theinhibitorysynapse is shownin thefigure5.6.Itis almost ident icalto the excita torysynapse withoutthe tran sistorsM3,M4 and the membrane capacit ance

e....

Thedrainof MI isgrounded,soitbecomes the source.Since allthesynaptic out puts will betiedtoget hertoconstit utethe activationbusforthe neu ron.

applicationof thepulsesatthe galeofMlwillresul tin withdrawalofcharge fromthe tota lmembranecapacitance. Thisdischarge current,howeverdepends on the weightvoltageVIOlat the galeof~12, Mostoftena strongerinhi bition (com paredto theexcitatio n) isrequired so thattheweight voltage willbearound 5volts,This will makeM2 operatein the linearregion.Sincethesourceof MI is grounded.Mlwillalsohe inthelinear region,

+1

Voltage builtup duetoa singlesynapses

2·

0.5 1.5

• :modeloutput -.: spire:output

2.S 3 3.5

Figure,';.5:Activationvoltage' due10one synapseusing(!(lllatioll~,S 13.,'i- InIUI II 5.1i.A spicesimulation isgiven for courparision.