HAL Id: hal-01222701
https://hal.archives-ouvertes.fr/hal-01222701
Submitted on 30 Oct 2015
HAL is a multi-disciplinary open access
archive for the deposit and dissemination of
sci-entific research documents, whether they are
pub-lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.
L’archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d’enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
retina: towards mixed synchronous/asynchronous
algorithms.
Antoine Manzanera
To cite this version:
Antoine Manzanera. Morphological segmentation on the programmable retina: towards mixed
syn-chronous/asynchronous algorithms..
6th International Symposium on Mathematical Morphology
(ISMM’02), Apr 2002, Sydney, Australia. �10.1.1.114.3040�. �hal-01222701�
PROGRAMMABLE RETINA:TOWARDSMIXED SYNCHRONOUS/ASYNCHRONOUSALGORITHMS
ANTOINEMANZANERA
E oleNat. Sup. deTe hniquesAvan ees 32,BdVi tor-75015Paris-FRANCE Antoine.Manzaneraensta.fr
Tel: 33145524442
Abstra t. Wepresentanewalgorithmto omputethewatershedtransformonamassively parallelma hine.Logi minimization,togetherwithanovelapproa htothenotionof dynam-i s,allowtheimplementationofthewatershedalgorithmona ellularautomatagrid,su h astheprogrammableretina. Wetheninvestigate thepro ta quiredfrom omputingsome partsofthealgorithminanasyn hronousway,intermsoftimeandenergy onsumption.
Keywords:watershed,programmableretina,dynami s,Bu on'smethod, syn hronous/asyn- hronousparallelism,SIMD,FIFO
1. Introdu tion
Implementationofmorphologi alalgorithmsonmassivelyparallelma hinesis nota hallengingissueinitself,sin ethetranslationinvarian eandlo al knowl-edge had alreadybeenset asprin iples of themorphologi aloperators by its reators [10℄. But on the onehand, the de eiving su ess of large size array SIMDma hines,dueto on eptiondiÆ ulties, ostoftheI/O ow,andla kof eÆ ien yof ertainoperations,andontheotherhand,thegreatprodu tionof thealgorithmi ists [9℄to optimizetheimplementation onsequential ar hite -tures,havepushedtheaÆnitiesbetweenmathemati almorphologyandSIMD intotheba kground.
Today,the onstantimprovementofCMOSte hnologiesallowstointrodu e aroughintelligen ewithineverypixelofanele troni amera,thusleadingto theintegration ofmoreandmorepowerfulprogrammableretinas (Se tion2). But unfortunately (resp. fortunately), the algorithmi problems remain dif- ult (resp. interesting), as we wish to illustrate in this paper through the exampleof thewatershed (Se tion3). First, be auseof theharsh onstraints duetothelimited apabilityofthepro essors,thatfor etoa onstante ortof logi minimization(Se tion4). Andse ond,be auseoftheinherentlimitation of the SIMD programming, that in iteto investigate the possibilities o ered byar hite turalsophisti ationsoftheSIMDmesh(Se tion5).
nan-ANALOG
PROCESSING
PHOTO−
TRANSDUCTION
DIGITAL
PROCESSING
A / D
CODING
V
s
V
t
(1) (2)Fig.1. (1)Thefourfun tionallayersoftheprogrammableretina. (2)TheNSIPdigitizing pro ess:time-amplitude odingforlevel-setspro essingandanamorphoses omputation.
2. The programmableretina
An arti ial retina is a ir uit ombining image a quisition and pro essing fun tions. The interest of su h devi e is to redu e dramati ally the ow of information within avision system,byperformingtheimagepro essing om-putationsin thefo al plane. Theprogrammableretina[5℄, [2℄,[8℄isaCMOS sensoranda ellularSIMDma hineatthesametime,withaverysmalldigital pro essor integrated into every pixel. Su h a ir uitis adapted to real-time onstrainedimage pro essingsystem,thankstoitshigh degreeof parallelism, aswell as embarkableorautonomous systems, thanks to its performan es in termsof ompa tnessandlow-power.
Figure1(1)shows(in1D)theorganizationofthea quisitionandpro essing fun tionsintheprogrammableretina. Theanalogpro essinglayeris onsidered hereasthesimplestpossibleanddedi atedtoveryearlyimagepro essing,su h asspatio-temporal ltering. Infa t,wewillsuppose thatthesignalpro essed by the digital part an be the luminan e image itself, but also a temporal di eren iation,oraspatialgradientintensity,bothfun tionsbeing omputable withsmallanalog ir uitry[7℄.
Ouralgorithmi interesta tuallybeginswiththethirdlayer:analog/digital onversion(ADC).ThroughtheNSIP(nearsensorimagepro essing)prin iple [4℄,avery ompa tADC anbeperformedundertheformofa apa itywhi h unloading timeis inversely proportionalto thevalueto measure. Theanalog valueobtainedfromphototransdu tionandanalogpro essing anthenbe dig-itizedthroughamultiple thresholding(Figure1(2)) thatwill produ easetof binaryimages. Fromamorphologi alpointofview,theseimagesrepresentthe levelsetsofthesignaltopro ess,that anbe odedinthedigitalmemory,orif possible,pro essedduring theADC.Furthermore,modifyingthenumberand
pro eduredilate4(p)f Booleanplanep1; p1=p N _p E ; p=p_p 1 _p SW 1 ; g pro eduredilate8(p)f Booleanplanep1; p1=p_p N _p S ; p=p 1 _p E 1 _p W 1 ; g
p
NW
p
SW
p
W
p
N
p
NE
p
E
p
SE
p
S
p
TABLEIRetinal odeofthebasi operatorsin4-and8- onne tivity. Ontheright,thenotations usedforthetranslations.Theerosionsareobtainedbyturningthe_(or)into^(and).
amorphologi alanamorphose.
Thedigitallayer,whi histhemainmatterinthispaper,ismadeofamesh of digital pro essors. It is an SIMD ma hine, with an important originality relatedtoalimited omputation apability,dueto atinymemory(onlyafew bitsperpixel). Thedataweoperateonareafewbinaryimages, alledBoolean planes(whi h anrepresentthegreylevelsofthesameimage). Theonly hard-wired operations on these data are: (1) One-pixel translation of one image. (2)Booleanoperationbetweentwoimages(or,and,xor,...).
With itsBooleanlanguage,theprogrammableretinaisnaturally ttedfor the non linear pro essing. TableI shows the omputation of the elementary morphologi aloperations,whi hare omputedinafewelementaryoperations anduse2bitsofmemory: 1bitofdata,and1bitforthe omputation.
Moregenerally,thetime omplexityofaretinal operatoris thenumberof elementaryoperationsthat omposeit, andthespa e omplexitythenumber of bits neededto perform it. Inourvery onstrained ontext ofmemory, the spa e omplexityisa riti alfa torfortheimplementationofalgorithms. The onsequen eis a onstante ortto redu e theamountof datamemory oded in thedigitalpro essors,in ordertomaximizethe omputationpower.
The omputationofthewatershed[3℄isasigni antexampleofthise ort. Indeed,thewatershed produ esabinaryimage, whi h anintrinsi ally repre-senta wide-dynami s signal, and thanks to time-to-amplitude oding, it an bepro essed\onthe y"onthein identsignalofthedigitallayer.
3. Computation of thewatershed transform
Fromnowon,wewillsupposethat the omputationismadeonadigital bidi-mensional signalI of rangevalue[0;n℄, knownby itsn binarythresholds, or levelsets,fI i g i2[1;n℄ ,withI i =fx2Z 2
;I(x)<ig,whi hareeitherreadonthe y from theADC, orprodu ed from an inner digital oding, thus o upying log
2
(n+1) bitsofmemory.
In the retinal algorithms presented below, the operations are performed within afully parallel framework overthepixels. The instru tionsare either elementaryoperations,orpro edurespresentedearlier. Theoperatorsusedare
pro edurere onstru tion(p;p 0
)f Repeatuntilstability,f
dilation(p); p=p^p 0 ; g g
pro edureopeningbyre onstru tion r (p)f Booleanplanep 1 ; p1=p; Repeatrtimesf erosion(p); g re onstru tion(p;p1); g TABLEII
Retinal odeofthegeodesi re onstru tion(left),andopeningbyre onstru tion(right).
pro edurehomotopi dilationEAST(p)f Booleanplanep 1 ,p 2 ; p1=:p^p E ; p2=p1_p N 1 _p S 1 ; p 1 =:p 2 ^p W ; p=p_p 1 ; g
pro eduregeodesi SKIZ(p;p 0
)f Repeatuntilstability,
andforXinfeast,south,west,northg (periodi allyrepeated)f homotopi dilationX(p); p=p^p 0 ; g g TABLEIII
Retinal odeofthehomotopi dilation(left,theotherdire tionsarededu edbyrotation), andofthegeodesi SKIZ(right).
pro edure willbemeasured,asthesumofthedatabits (theparameters)and the omputation bits (the \lo al" variables of the pro edure, in luding the hiddenvariablesusedbysub-pro edures).
The pro edures in this se tion anbepresentedbrie y, asit is a lassi al implementationofthewatershedbyimmersion simulation. Itisbasedontwo binaryfun tions: thegeodesi re onstru tionandthegeodesi SKIZ.
Thegeodesi re onstru tionis omputed bytheleftpro edure ofTableII. It uses 3 bits of memory. The right pro edure of Table II shows a dire t appli ationin morphologi al ltering: theopeningbyre onstru tionbyaball of radiusr. This pro edure,that needs3bitsof memorytoo,will beused in thenextse tion.
The lassi al approximationof the SKIZ by the dual homotopi kernelis omputedthanksto therelaxationof thehomotopi dilationoperator,shown on Table III (left), that uses3bits of memory. Thegeodesi SKIZ is simply obtainedby omputingalogi alandwith theimage ofreferen e,so ituses4 bitsofmemory.
Finally,thewatershedofthedigitalsignalIofrangevalue[0;n℄is omputed thankstothepro edureofTableIV(left),where pisthebinaryoutput. This pro edureuses4bitsofmemory. Toover omethewell-knownproblemof over-segmentation (Figure2(2)), a lassi alsolution onsistsin usingan imageof markers(Figure2(3))to onstrainthetopologyofthe nalwatershed(Figure2 (4)). The onstrainedpro edure isshownonTableIV(right),wheremis the
pro edurewatershed(I;p)f Booleanplanep 1 ,p 2 ; p=I[1℄; Fork=2tonf p1=I[i℄; p 2 =p; re onstru tion(p2;p1); p 2 =:p 2 ^p 1 ; p=p_p 2 ; geodesi SKIZ(p;p 1 ); g g
pro edurewatershedbis(I;m;p)f Booleanplanep 1 ; p=m; Fork=1tonf p 1 =I[i℄; geodesi SKIZ(p;p1); p=p_m; g g TABLEIV
Retinal odeofthewatershedtransform(left),andofthewatershedwith mark-ers(right).
( 2 )
( 3 )
( 4 )
( 1 )
Fig.2. Resultsof thewatershed transform. The omputed signalinthisexampleisthe gradientintensityofImage(1). Therawwatershedtransformisshownonimage(2). The markersarethe(white) onne ted omponentsofimage(3):onediskmarksthewhitetaxiat the enter,anotherdiskmarksthedark arontheleft,there tanglemarkstheba kground. Thewatershed omputedwiththesemarkersisshownonImage(4)(apost-pro essisused toeliminatethe urvebetweenthetwo at hmentbasins).
It an beseenthatthe omputationofthewatershedlends itselfnaturally to a ompa timplementation, welladapted toourar hite ture. Nevertheless, the markersneed tobe omputedbefore, and in most asesof segmentation, theyarenoteasyto nd. Anotherwaytoavoidover-segmentationistodis ard non-signi antminima. Thisisdonethanksto lteringanddynami ssetting.
4. Filtering and dynami s setting
Indeed, non-signi ant minima anappear for tworeasons: (1) noise on the signal, and (2) poor-dynami s at hment basins. The noise problem is ad-dressed through morphologi al ltering (see Figure 3): let be the minimal thi knessofabasin admissible,the ltering onsistsinperforminganopening byre onstru tion byaball ofsize oneverylevelsetof I. This anbedone withoutextra-memory.
Thedynami sproblemismore riti al. The lassi alsolution[6℄toremove minima of dynami s inferior to h (basins of depth inferior to h) onsists in
1
2
ρ
ρ
Fig.3. The rst auseofover-segmentationinthewatershed: noiseonthesignal. Basins ofthi knessinferiorto(1)aresuppressedthankstomorphologi al ltering(2).
binaryre onstru tionof :I i
in :I i h
,foreverylevelsetsu hthati>h. But this method implies that twolevelsets distant of hmust be available in the memory at the same time. If the signal is not already oded in the digital memory,this ostsh 1bitsofmemory. Thesolutionproposed inthispaper addressesdi erentlythedynami sissue,andusesonly1extrabit ofmemory.
2
3
4
1
h
h
Fig.4. These ond auseofover-segmentationinthewatershed: poordynami sminima. Basinsofdepthinferiortoh(1)aredis ardedthankstotheBu on'smethod(2),(3),(4).
The prin ipleofour solution(see Figure4) is to omputetwowatersheds ofthesamesignal. Thelevelsets ofthesignalaresub-sampledwitharegular step of h,and two onse utivelevel sets are used for omputingtwodistin t watersheds. So the two watersheds are omputed over a sub-sample of the levelsets, witha regularstep of 2h, in su h awaythat the twosub-samples are in phase quadrature (see Figure 4(2) and (3)). We then postulate that theonly signi ant urvesof thesegmentation arethose that appearin both watersheds, and then the resultsis omputed by performing alogi al \and" betweenthetwowatersheds(seeFigure4(4)).
This\Bu on's"methodisindeedbasedonthemonodimensionalversionof theneedleexperien eofthefren hnaturalist. Throwingrandomlyasegment of lengthl (the needle) overaline marked by samplesregularly spa ed of h, what is theprobabilitythat the segmenthits twosamples? It is easyto see thatitisthe ontinuouspie e-wiseaÆnefun tionoflthatisequalto0ifl<h,
pro edurewatershedter (;h; ) (I;p)f Booleanplanep 1 ,p 2 ,p 3 ; p=I[h℄; openingbyre onstru tion (p); p1=I[2h℄; openingbyre onstru tion (p 1 ); fori=3tod n h ef p2=I[ih℄; openingbyre onstru tion (p 2 ); if(i%2==0)f p 3 =p; re onstru tion(p 3 ;p 2 ); p3=:p3^p2; p=p_p3; geodesi SKIZ(p;p 2 ); gelsef p3=p1; re onstru tion(p3;p2); p 3 =:p 3 ^p 2 ; p1=p1_p3; geodesi SKIZ(p1;p2); g g fori=1tof erode4(p); g p=p_p1; SKIZ(p); g TABLEV
Retinal odeofthewatershedwithparameters(;h;).
meansthat:
Nominimumofdynami sÆ<h anappearin thewatershed. All minimaofdynami s Æ>2hwillappearinthewatershed.
Amimimumofdynami s h<Æ<2hwillappearin thewatershedwitha probability
Æ h h
.
Infa t,thea tual ombinationofthetwowatershedsisabitmorediÆ ult than a simple and. Indeed, the quantization of the level sets dis riminates minimaofpoordynami s,butalsoindu esashiftonthefrontiersoftheother minima, i.e. theirpositions anvaryin thetwowatersheds. If isthe maxi-malshiftadmitted,W
1 andW
2
thetwo\semi-watersheds",theinterse tionis omputed as: W 1 ^(W 2 B
), where isthedilation andB
adigitalball of radius . The value of the shift depends on (1) the quantization step h, (2) thelo alsharpness ofthe peak in the signal. So theparameter will be hoosena ordingto theminimalsharpnessof a ontourdesired,and propor-tionnallytoh. Furthermore,theright ombinationisnotthe ommonpoints, but the ommon losed urvesof thetwo watersheds, so the nal watershed is W =HK(W 1 ^(W 2 B
)), where HK is the homotopi kernel. Seeend ofpro edureofTableV: asweareworkingon omplementarywatersheds,we omputeinfa t W =SKIZ(W
1 _(W 2 B )).
Aresultof omputationisshownonFigure5. Theimportanta hievement of this method is that omputing the watershed with ltering and dynami s setting anbeperformed ata onstant memory ost, independantof the pa-rameters: 5bits.
5. Bene tsof asyn hronism
Wehaveshownthatthewatershedtransform,withitsre nementsusedin mor-phologi alsegmentation, anbeimplementedataverylowmaterial ost(and, or, not, translation, 5bits of memory), on a ellular automata grid. This meansthat,today,it anbe omputedwithinthepixelsofadigital amera,at
( 4b )
( 5 )
( 4a )
( 0 )
( 1 )
( 2 )
( 3 )
Fig.5. Resultsofthewatershedwithparameters. (0)Originalimage(200x200x8bits). (1)Rawwatershed( =0;h =0). (2)Filteringwithnodynami ssetting (=2;h=0). (3) Dynami ssetting withno ltering ( = 0;h = 8). (4a) and (4b) Dynami s setting byBu on's method: two lteredwatersheds ( = 2) are omputed bysub-sampling the greylevels with a step = 16, su h that the two sub-samples are inphase-quadrature. (5)Theresultisobtainedby ombinationofthetwowatersheds(=2;h=8;=2).
Indeed, by performingthe segmentation dire tly in the sensor, weaim at saving timeand energy, thanksto thein-site massivelyparallel omputation. However, if we examine the eÆ ien y of the SIMD parallelism used in the previouspro edure,itturnsoutthat the omputationof relaxationoperators likethe geodesi re onstru tion anbeveryineÆ ient. Inthe typi al aseof there onstru tion of a urvefrom oneof its points,only afewpixels hange theirvalueateveryiteration,thentheSIMD omputationspeedisnotbetter thanthesequentialone,andtheenergywasteismu hgreater!
Thisdrawba koftheSIMDparallelism anbeminimizedwithinthedigital layerof the programmable retina, thanksto asyn hronouspropagation. The useofsemi-stati shiftregistersasdigitalmemories[1℄makespossiblethe fol-lowingasyn hronousphenomenon: asetof\a tive"pixels opytheirvalueon their neighbors, that be ome a tivetoo, and so on until stability. This sim-ple asyn hronousfun tion be omeveryinterestingifthe onne tionsare pro-grammable. Indeed,itallowsthe omputationofthegeodesi re onstru tionin onerelaxedasyn hronouspropagation(see Figure6(1)). This omputationis faster,be ausethepropagationtakesmu hlesstimethanthesyn hronous (di-lation+interse tion): the asyn hronous propagation time has been estimated [1℄to 1.5nsperpixel,whi h orrespondtothe(unrealisti ) ontrolfrequen e of 12GHzin syn hronous. And itis mu h moreeÆ ient, be auseduring the propagation,onlythea tivepixels/pro essors onsumeenergy.
Whatarethealgorithmi impli ationsofthisasyn hronousparallelism? In fa t,itismorethanasimplee onomi alwaytogetthere onstru tion,be ause,
( 1b )
( 2a )
( 2b )
( 1a )
Fig.6. Useofanasyn hronousparallelismonaprogrammabletopology. (1a)Thegeodesi re onstru tiononastati meshisobtainedthroughiterateddilation/interse tion. (1b)On a programmablemesh, after disablingthe onne tions to ba kground (bla k) pixels, the re onstru tion isobtainedthroughone asyn hronouspropagation fromthe markerpixels. (2a) When omputingtheSKIZon astati mesh,thesyn hronousalgorithmbe omevery ineÆ ientassoonastheba kground(bla k)be omeonepixelthi k. (2b)Onaprogrammable mesh,afterdisablingthe onne tionstoforeground(white)pixels,andtomultiplepointsof theba kground,there onstru tionfromtheextremitiesoftheba kgroundallowstoremove theopen urvesinoneasyn hronouspropagation.
anbe onstrainednotonlybytheimageitself, butbytheresultofanSIMD operator overthe image. Figure 6(2)shows anexample for theSKIZ, whi h is the other relaxation operator used in the watershed omputation. In our system, theSKIZisthen omputedin twophases: onesyn hronous,and one asyn hronous.
Moregenerally,theasyn hronousoperations anbealgorithmi allyseenas animplementationofalimited lassof FIFOs. TheFIFOdatastru tureand algorithmi s[11℄isofspe ialinterestforus,be auseits omputing apabilities are, in some measure, diametrally opposed to the apabilities of the SIMD ma hine, and then the oexisten eof the twosystems on thesame ir uitis verypromising.
6. Con lusions
On the base of the two last ir uits design [2℄, [8℄, the next generation of programmable retinawill havebetween15and 20bits of digitalmemoryper pixel,a ontrolfrequen eadjustablebetween1and100MHz,andaresolution of200x200pixels. Theexpe ted omputationtimehasbeenestimated(see Ta-bleVI)ontheimageofFigure5(0),whi hisan8bitsimageofsize200x200,for a ontrolfrequen eof10MHz,andforthetwomodes: syn hronous,andmixed syn hronous/asyn hronous. Obviously, themixed mode ismoreeÆ ient,and inaddition,unlikethesyn hronousmode,whose omputationtimeisstrongly dependent of the data, the time of the mixed mode is not mu h di erent in theworst ase,be ausetheasyn hronouspropagationtimeisalmostnegligible withrespe tto thesyn hronousperiod.
paral-watershed(I;p) watershedter (2;8;2) (I;p) Syn hronous 1.4ms 1.9ms MixedSyn /Asyn 0.3ms 0.5ms TABLEVI
Estimationofthe omputationtime.
ingtheir omputation apabilities,fromthelo alto theregional. Su h mixity is already possible at alow material ostand will probably exist in the next generation of programmable retinas, in whi h the algorithms presented here willbeeasilyimplemented. Now,otherwaystoexploittheasyn hronous prop-agationsshould beexplored. Inparti ular,howto onstrainthepropagations by something \smarter" than the simple topology, typi ally lo al on gura-tionsthatappearduringthepropagation,thusobtainingtheimplementations ofmore lassesofFIFOs? Futureworkswilladdressthisissue.
A knowledgments
The author wish to thank Thierry M. Bernard and Mi hel S hmitt for their ontributionto thiswork.
Referen es
1. T.M.Bernard.Multi-purposesemi-stati shiftregistersfordigitalprogrammable reti-nas. InPhot.WestConf.onSCSSIA,volume3965.SPIE,2000.
2. T.M.Bernard,B.Y.Zavidovique,andF.Devos. AprogrammableArti ialRetina. IEEEJSSC,28-7:789{798,1993.
3. S.Beu herandF.Meyer. TheMorphologi alApproa htoSegmentation: The Water-shedTransformation. EdwardDoughertyed.,1991.
4. J.-E.Eklund,C.Svensson,andA.astrom.VLSIimplementationofafo alplaneimage pro essor-arealizationoftheNSIP on ept. IEEEVLSI,4-3:322{335,1996. 5. P.Garda, A.Rei hart, H.Rodriguez, F.Devos, and B.Y.Zavidovique. Une retine
ele troniqueautomate ellulaire. Traitementdusignal,5-6:435{449,1988.
6. M.Grimaud. Anewmeasureof ontrast: the dynami s. InImageAlgebra and Mor-phologi alPro essingIII,volume1769,pages292{305.SPIE,1992.
7. A.Moini. VisionChips.KluwerA ademi s,1999.
8. F. Paillet, D. Mer ier, and T. Bernard. Se ond generation programmable arti ial retina. InPro .IEEEASIC/SOCConf.,pages304{309,1999.
9. M.S hmittandL.Vin ent. Morphologi alimageanalysis.Apra ti alandalgorithmi handbook. CambridgeUniversityPress,toappear.
10. J.Serra.Imageanalysisandmathemati almorphology.A ademi Press,London,1982. 11. L.Vin entandP.Soille.Watershedsindigitalspa es:AneÆ ientalgorithmbasedon
