Countering illumination variations in a video surveillance environment

Countering Illumination Variations

in a Video Surveillance Environment

R. Costantini,G. Ramponi a

, J. Bracamonte, B. Piller b

, M. Ansorge,and F. Pellandini

Inst. of Microtechnology, Uni. of Neuch^atel, CH-2000 Neuch^atel, Switzerland

a

Uni. of Trieste, I-34127 Trieste, Italy

b

SysNova S.A., CH-9525Lenggenwil, Switzerland

e-mail: [email protected]

ABSTRACT

In theeld of video technology for surveillanceapplications it is often necessaryto copewith the phenomenon of

illuminationvariations. Infact,ifnotcompensated, suchvariationscanfalselytriggerthechange detectionmodule

thatdetectsintrusionsinvideosurveillancesystems,thusaectingtheirreliability. Manystudieshavebeenmadeto

solvethechangedetectionproblemundervaryingilluminationconditions. Mostofthepublishedmethods,however,

relyonlyontheluminanceinformation. Thealgorithmproposedinthispaperexploitsindependentlytheinformation

of each band ofthe RGB colorspace of thevideo sequences, thus producing achange detection algorithm that is

morerobusttoilluminationvariations. Theseilluminationvariationsaregloballymodeledbytheso-calledVonKries

model (alsoknown asdiagonal scaling model). This model isgenerallyusedto solvethecolorconstancyproblems,

whereconformancetoareferenceimageilluminationhastobeguaranteed,likein colorimageretrievalapplications.

The useof this model is motivated by its lowcomputational costand by the interestof studying the relationship

between colorconstancy andchange detection. Based on practicalexperiments which conrm the interest in this

method, new and more robust change detection algorithms are expected to be designed. In addition, the paper

proposestheuseofaniterativeschemewhoseaimistoimprovetheresultsobtainedinthechangedetectionmodule,

andwhichisindependentofthismodule,i.e.,itcanbeusedwithotherchangedetectionschemes. Itwillbeshown

thattheiterationcanimprovethequalityofthenalchangemask,thuspermittingtoobtainamoreeectivechange

detectionscheme.

Keywords: Changedetection,video-surveillance,colorconstancy.

1. INTRODUCTION

Withintheeldofvideosurveillancedierentsituationsofpotentialdangercanoccurandhavetobesignaledbythe

surveillancesystem. Thenatureofthesesituationscanvary,butfromthepointofviewofthesurveillancesystemit

canbesimplyrepresentedastheoccurrenceofachangein animageofthevideosequencewithrespecttoanimage

takenasareference. Fromonesidethischangecanbeduetotheintrusionofanunknownpersoninaroom/building,

and so,it can be related to thepresence of motion in ascene that, normally, is supposed to remainstatic. From

theothersidethechangecanderivefrom thepresencein thecurrentimageofanewobjectwhichdoesnotappear

in thereferenceimage andis notforeseenin that particularcontext, orremainstoolongat thesameplace (asfor

examplesuspiciousbagsleft in airportsorrailwaystations);in thiscasethechange thatshould besignaledbythe

surveillancesystemisnotrelatedtomotion,buttothepermanenceofsomethinginthescene. Eitheritderivesfrom

motionorpermanence of objects/persons in ascene, the change which is produced in the imagesconstituting the

video sequence hasto be detected and signaled. This task can be accomplishedby one ormorechange detection

algorithmsembedded inthesurveillancesystem.

The phenomenon of the variationof illumination that could occurin avideo sequence can signicantlyaect the

performanceof changedetectionalgorithms. Infact,such avariationcouldbedetectedasachangecaused bythe

presenceof aperson in thescene, thus constitutingwhat is called afalse alarm. Since it ispreferableto havethe

lessfalsealarmsaspossible,theilluminationvariationshavetobetakenintoaccountandcompensated.

Generally it is quite diÆcult to make a classication of the possible cases of illumination variations, since such

variationscanbeverydierentonefrom theother. Theycanbeproducedbothinoutdoorscenes,where theycan

becausedbythechangeofthesunlightilluminationcausedbythemovementofacloudoverthesceneforexample,

Copyright 2001 Society of Photo-Optical Instrumentation Engineers (SPIE). This paper is made available as an electronic reprint with permission of SPIE. One print or electronic copy may be made for personal use only. Systematic or multiple reproduction, distribution to multiple locations via electronic or other means, duplication of any material in this paper for

a fee or for commercial purposes, or modification of the content of the paper are prohibited.

totreatallthecaseshasnotyetbeenfound.

Themethodproposed inthepresentpapercoverssomesignicantcases,thatcanberepresentativeofcommonreal

worldsituations. Furthermore,theproposedmethodpermitstodeepentheanalysisoftherelationshipbetweentwo

distincteldsoftheimageprocessingdisciplinelikecolorconstancyandchangedetection.

InSection2adescriptionofseveralchangedetectionalgorithmsisprovided,whereastheso-calledVonKriesmodel

ispresentedinSection 3. Section4isthen discussingtheproposedalgorithm,thecorrespondingexperimentaltests

and achieved resultsprovingtheirrobustnessbeingthoroughly presentedin Section 5. Finally, the conclusionare

drawnin Section6.

2. CHANGE DETECTION ALGORITHMS

The methods which specicallydetect changes due to the presenceof motion in ascene are usuallyreferred to as

motiondetectionalgorithms,andtheirtaskisnotonlytosignalthechange,butalsotoprovidesomesupplementary

informationaboutthemotion,asthedisplacementvectorsorthevelocityeld. Manytechniqueshavebeendeveloped

to accomplish this task. Ina great numberof algorithms an estimate of the motionelds is found by computing

theoptical ow oftheimages formingthevideosequence. Sincetheoptical owis denedastheapparentmotion

of brightnesspatterns in these images, it generallycorrespondsto the motioneld, but in somecasesit canbring

tofalseestimation,likeinthewell-knownbarberpolesequence[1],wheretherealmovementofthepoleiscircular,

whiletheperceivedopticalowisvertical. Excludingthespecialcases,likethelatterone,these methodsarequite

eectiveand usually producegoodresults. Other methods makeuseof aminimization strategyof acost function

to nd the displacement vectors,as it is commonly done in the motionestimation block of atypicalvideo coder.

An evaluation of some motion detection techniques can be found in [2], while in [3],[4],[5] some new algorithms

areproposed. Acharacteristiccommontoall themotion detectionalgorithmsistheir computationalburden,even

thoughmuch workhasbeendonetoreduceit.

Thepresentworkis notinterestedintheproblem ofmotiondetection, sincetheaimsoftheproposed algorithmis

to obtainachange detectionmaskand notamotioneld. Forthis reason,themotion detectionmethods willnot

befurther described.

The design of eective change detection algorithms has been of interest in many studies in the image processing

eld. Atthemomentthere existseveralmethods thataresuitableforthedetectionofchangesin avideosequence,

and that dierone from the other according to the change/noise model they use and thedistinct strategiesthey

apply. A rstclassicationof thechange detection algorithmscanbefoundin [7]. Inthis work twonew methods

areproposed: theDerivative Model Methodandthe ShadingModel Method. Mainly, thesetwomethods havebeen

designedto copewith the phenomenonof illumination variation, andit is shown that theShading ModelMethod

is particularly robust even to strong variations of illumination. This method is based on the model of intensity

valueproposed byPhong[6], where theintensityvalue I

p

ofapixel in an image ismodeledasthe productof the

illuminationI

i

andaso-calledshadingcoeÆcientS

p :

I

p

=I

i S

p

: (1)

Inordertoestablishifapixelinposition(i;j)ofanimagehaschanged,thealgorithmcomputesthevarianceofthe

ratiosofthepixelintensityvaluesin anNN windowcentered atthepixel'sposition:

VAR

W

I

c

I

r

= 1

N 2

X

(i;j)2W I

(i;j)

c

I (i;j)

r

!

2

(2)

whereWrepresentsthesetofpixelsinthewindowofsizeNN,I (i;j)

c

andI (i;j)

r

aretheintensityofthepixel(i;j)

belongingtothecurrentandreferenceimagesrespectively,andisthemeanvalueoftheratiosI (i;j)

c

=I (i;j)

r

withinthe

window. Ifthevarianceoftheratiosexceedsacertainthreshold,thenthepixelisclassiedaschanged,otherwiseit

isconsideredasunchanged. Thismethodexhibitsgoodperformancesevenwithstrongilluminationchanges. Thanks

toitseectiveness,otherandmorerecentworkshavebeenbasedonthismethod,asthoseproposedin [9]and [10].

The shortcoming of this method is that many division operationshave to be computed, resulting in a quite high

computationalcost.

Analternativetechniqueisproposedin[8],whereinsteadofcalculatingthevarianceoftheintensityratiosofthe

images,achangedetectionisperformedusingsomecircular-shiftmomentsofthepixelsintensitywithinaspecied

window. It is demonstratedthat this method requires asmaller numberofoperationsper pixel thantheprevious

method, still assuring the same quality of results. The algorithm proposed in [8] belongs to the broad class of

statistical change detection algorithms, which constitute a great portion of the change detection algorithms, both

becauseoftherelevantstudiesthat havebeencarriedonandfortheresultsthattheyallowtoobtain.

Thestatisticalchangedetectionalgorithmsareallcharacterizedbythesameprocessingoftheinformation. Inarst

phasesomeassumptionsaremadeaboutthestatisticsofthenoisepresentintheimage,then,toevaluatethepossible

change ofapixel,adistance functionis calculatedbetweenapixelin thecurrentimage andthecorrespondentone

in the referenceimage, generallyconsidering the valuesof theother pixels in the neighborhood, i.e., ona window

centered at the pixel's position. Thestatistical properties of this function are then studied, in order to predicta

rangeof valuesthatare assumedin thecasethepixel hasnotchanged. Asubsequentthresholdmechanismallows

todistinguishchangedpixelsfromunchangedones.

In[11]thefunctionsthat areproposedarethefollowing:

i

1

= X

k 2Wi jI

k

r I

k

c j or

i

2

= X

k 2Wi (I

k

r I

k

c )

2

(3)

whereW

i

representsthewindowaroundthepixeli,I k

r andI

k

c

representrespectivelythereferenceandcurrentvalue

ofthepixelsintensityinthatwindow,andisanormalizationparameteradjustabletodierentnoiselevels. After

thethresholdstep,whichisbasedonstatisticalevaluations,achangemaskisobtained. Sincethismaskcanpresent

holesandroughcontours,thealgorithmusesinadditionaMarkovRandomFieldreningtechnique,whichconsists

of imposing theminimization of a costfunction obtainedfrom the maskby some reasonableconsiderations. The

nal resultis quite good, since thechange maskis accurateand wellrened. However,this method hasnot been

designedspecicallyforchange detectionin presenceofilluminationvariations.

Ontheotherhandin [12]thefunctionproposed is:

F

i

= 1

N X

k 2W (d

k

i )

4

(4)

where d

k

= (I k

r I

k

c

), and

i

is the mean value of d

k

within W. It can be easily seen that this function is the

fourth-ordermomentof the dierencesbetweenpixels intensities in the chosenwindow. This algorithm takesinto

considerationalsoothercharacteristicsoftheimages,thusobtainingahigh-order-statistic(HOS)test. Theresulting

detectionmask,asinthepreviousalgorithm,isfurtherprocessedinordertoobtainmorecoherentandhomogeneous

regions. Inthiscase,somemorphologicaloperationsareperformed. Theresultsobtainedaregood,butthismethod,

likethelastonepresented,hasnotbeendirectlydesignedforapplicationsinwhichtheilluminationcouldsuddenly

vary. Tocopewiththeproblemofilluminationchanges, morespecic algorithmshavebeendesigned.

In[13] analgorithm isproposed which applies thestatistical change detectionmethod described in [11] onthe

reectancecomponentoftheimage,i.e.,thecomponentwhichiscalledS

p

inEq.1,insteadofitsintensity. Sincethe

reectancecomponentofanimagerepresentstheinnerpropertiesoftheobjectscontainedintheimage, andisthus

notinuencedbytheillumination variation,theresultingchangedetection algorithmis morerobustto changes of

illumination. Thereectance componentis extractedfrom theintensityimage bytheuseof ahomomorphiclter.

Thewholealgorithmguaranteesbetterperformancewithrespectto thesimpleoriginalchangedetectionalgorithm,

sincethereectanceimage islessdependentonchanges ofillumination. Obviously,the homomorphicltercannot

exactly separatethereectanceandtheilluminationcontributionsthatconstitutetheimage intensity,andso,only

illuminationvariationsthatdonothavehighspatialfrequenciescanbeeliminated. Thiscorrespondstothesituation

ofaquiteuniformilluminationchange.

In[14] and[15] adierentstrategyis appliedto copewiththe phenomenonofthe illuminationvariation. Inthese

twoworks the referenceimage is updatedover thetime, such that the illumination variationcan be included and

thuscompensated. In[14]thisupdateisdonebyacontinuousrefreshofthereferenceimageinfunctionofthespeed

ofthechangeofillumination,whileaKalmanlterisusedin [15].

Themethodsthathavebeenpresentedareallbasedontheintensityvaluesoftheimages. Ifthechangedetection

hastobeperformedoncolorimages,usuallydescribedintheRGBcolorspaceoftheacquisitiondevices,itispossible

totheYUV space. Butthereexistalso somechangedetectionalgorithmsthatexploit theinformationgivenbythe

colorcomponentsandwork directlyontheRGBcolorspace.

An interestingexampleofamethod of thiskindcanbefoundin [16]. Theaimof thealgorithm isto separatethe

shadowsproducedbythepresenceofapersonin ascenefromthepersonitself, and,tosomeextent,tocompensate

foraglobalilluminationchangethatcouldaectthescene. Theconceptofthisalgorithmisto computeadistance

betweenthecolorpixelsthatformthecurrentimageofthesceneandtheirexpectedvalues,whichhavebeenobtained

byapredictionmechanismthatisbasedonthestatisticalpropertiesofthebackgroundwhichisxedandconstitutes

thereferenceimage. Ifthisdistanceexceedscertainthresholds,whicharestatisticallyadaptedframebyframe,then

achangeissignaled. Thismethodappliesastrategythatissimilartotheoneproposedinthepresentpaper,andit

ensuresoptimalresultswhendealingwithshadowsandglobalilluminationvariations. Anyway,themethodproposed

in[16] wasnotdesignedtoberobusttostrongilluminationvariations,eventhoughtheresearchisonprogress.

ThemethodproposedinthepresentpaperconsistsofachangedetectionintheRGBcolorspaceandmakesuse

ofasimplemodelforthechangeofilluminationthatcouldoccurinthescene. ThismodeliscalledDiagonalScaling

Model, andwillbedescribedinthefollowingsection.

3. THE DIAGONAL SCALING MODEL

Let E(~x ;) be the intensity of illumination of wavelength given by asource E and arriving at aposition ~x on

asurfacepresentin a scenelmed by avideocamera. Theactual pixelvaluesmeasuredby thek-th sensor ofthe

cameraare:

k (~x )=

Z

E(~x ;)[S

s

(~x ;)+S

b

(~x ;)]R

k

()d withk=1;2;3 ; (5)

where R

k

() is thespectralresponse function ofthe sensor, and S

s

(~x;) andS

b

(~x ;) are respectivelythesurface

and body reections (also known asspecular and diuse reections) of the surface. Eq.5 represents the so-called

dichromaticreectancemodel[17]forthecolorvaluesperceivedbyavideocamera. Ascanbenoticed,theperceived

valuesdepend on thesensor spectralresponse,the illumination, and the inner characteristicof the surface,which

isrepresentedbytwofactorsS

s

(~x ;) andS

b

(~x;),thusexplainingthereasonwhythemodelis calleddichromatic.

The surface reection is dominant in metallic surfaces, where the incoming light is reected almost maintaining

its spectral content, so the dependence on is quite weak, while for dielectric/matt surfaces the body reection

representsthemajorcontributiontotheperceivedcolorvalues,sincethelightis almostcompletelyscattered inall

directions.

IftheilluminationchangesfromE(~x ;)toE 0

(~x;),thentheperceivedcolorschangeaccordingtothetransformation:

k (~x )=

Z

E(~x ;)[S

s

(~x ;)+S

b

(~x ;)]R

k

()d !

0

k (~x )=

Z

E

0

(~x;)[S

s

(~x ;)+S

b

(~x ;)]R

k

()d : (6)

Therelationshipbetween

k

(~x )and 0

k

(~x )isnotevident,butifsomeassumptionsaremadeaboutthecolorsensors

propertiesandtheilluminationsources,thenitcanbeexpressed inasimplerform.

In fact, it can be simply demonstrated that, if the spectral response of the sensors were a perfect Dirac impulse

Æ

k

(),then therelationshipbetweentheperceivedpixelcolorswouldsimplybe:

0

k (~x )=a

k (~x)

k

(~x ) (7)

where a

k

(~x) = E 0

(~x)=E(~x ), i.e., the color response that we have under the illuminant E 0

(~x;) would be simply

obtained by the previous one with a separated scaling of each color component. This can be expressed in the

followingway:

2

4 R

0

G 0

B 0

3

5

= 2

4 a

R

0 0

0 a

G 0

0 0 a

B 3

5 2

4 R

G

B 3

5

(8)

wheretheknownRGBcolorspaceisusedinsteadofthenotation

k .

Thisrepresentationofthechangeofthecolorvaluesobtainedbyavariationoftheilluminantofthescenecorresponds

totheDiagonalColorScalingModel,alsoknownasVonKriesmodel. Thisisoneoftherstmodelsthathavebeen

avisionsystemto thevariation ofthecolorof ascenecaused byachangeof theilluminant. Tosomeextent, this

propertyis naturallypresentin thehumanvisionsystem,whileithasto beembeddedin amachinevisionsystem.

Solving the colorconstancy problem corresponds to obtaininga representation of the scene which is independent

of the illuminant. The Von Kries model allows to implement this property, since it is possible to pass from an

image takenunder anunknown illuminantinto the sameimage under adierentand known illuminant, usingthe

coeÆcientsa

R ,a

G , anda

B .

Forrealimages,thespectralresponsesofaCCDorCMOScameraarenotexactlyrepresentableasDiracfunctions,

butinanycase,thismodelcaneectivelyrepresentthecolortransitionderivedfromachangeinillumination. Thisis

furtherprovedin[18],whereitisdemonstratedthatforthecasewheresurfaceilluminationcanbeapproximatedwith

a3-dimensionalbasisandreectancewitha2-dimensionalbasis(Maloney's2-3restrictions),then, independentlyof

theform ofthespectralresponse ofthesensor, thecolorconstancytransformationcanbeexpressedbyadiagonal

matrix. Thisisafundamentalresult,sincemanyscenesarewellrepresentablewithintheMaloney'srestrictions,and

sothediagonalmodeltsagreatnumberofreal situations.

The diagonal scaling model has been widely used for solving the problem of color constancyin many algorithms

where thiswasanecessaryconditionto guaranteegood results. Forexample thismodel isused in [19] inthe eld

ofimageretrieval,whereachangeoftheilluminationinthequeryimagecanavoiditsretrievalinagivendatabase,

whereallimagesaresupposedto beobtainedunderthesameilluminant.

Inthealgorithmproposedinthepresentpaper,thismodelofcolortransitionwillbeusedtomakeapredictionofthe

valueofthecolorcomponentsof apixel inthecurrentframeofavideosequence, obtainedfrom thecorresponding

valuesin areferenceframe. Thanks to this predictionit will be possible to establishif this pixel hassignicantly

changedbycomparingitspredictedandactualvalues. Suchpredictionscorrespondtoanormalizationofthecurrent

imagewithrespectto theilluminantofthereferenceimage.

4. DESCRIPTION OF THE ALGORITHM

4.1. The concept

Detectingachangein animage I

n

with respect toareference imageof avideosequenceis equivalentto ndinga

binaryimageM

n

such that:

M

n (i;j)=

1 ifI

n

(i;j)ischanged

0 otherwise

(9)

whereI

n

(i;j)indicatesthepixelofposition(i;j)oftheimageI

n

. Thechangein position (i;j)canresultfromthe

presenceof an object/person in thescene, from an illuminationvariation, and/orfrom the presenceof noise. The

proposed methodaimsatcreatingabinarymaskthatcorrespondsto thechangecausedbytheobject/persononly,

andisthusrobusttoilluminationvariationsandnoise.

Theconcept of the algorithm isto nda prediction of the valuesof thepixels in thecurrentframe, based onthe

values of thepixels in the referenceframe and onan estimate of the global colorvariations of the current frame.

Oncethispredictioniscomputedforeachpixel,thenadistancebetweenthispredictionandtheactualvaluesofthe

pixelisevaluated,enablingtheselectionofchangedpixelsfromunchangedonesinasuccessivethresholdingphase.

Tounderstand this concept, let us suppose that apossiblechange of illuminationin thecurrentimage of a video

sequencealtersthevaluesofthepixelsexactlyaccordingto thediagonalscalingmodel. Inthiscase,thetransition

oftheRGBcolorcomponentsofthepixelsin thecurrentframewould bespeciedbyEq.8,i.e., each pixelaected

onlybyachangeofilluminationwouldhavethefollowingcolorcomponents:

R i;j

n

=a n

R R

i;j

0 G

i;j

n

=a n

G G

i;j

0 B

i;j

n

=a n

B B

i;j

0

(10)

where n indicates the frame number. On the contrary, the pixels aected by a change due to the presence of a

person/object in the scene would have values dierent from those indicated by Eq.10. By computing a distance

between the actual values of the pixels in the whole image and their prediction found using Eq.10, it is possible

to decide uponpixel changes. The parametersa n

R , a

n

G , and a

n

B

, which are notknowna priori, could befound by

considering apixel which is notchangedin the currentimage, dividingits RGB color componentsby those ofthe

correspondingpixelin thereferenceimage.

Inpractice, forreal imagesthis method hastwoproblems. First, thediagonalscaling model doesnotperfectly t

a change of illumination that could occur in real images. Infact, this model does not consider the shadows and

a

R ,a

G ,anda

B

accordingtothepixel position: highervaluesforpixelscorrespondingto highlightedzones,smaller

values forpixels in shadowed zones. This shortcomeof the model will be partially compensated by the useof an

adequatedistancemetricandathresholdstrategy,stillmaintainingthestructureofthechangedetectionalgorithm,

i.e.,theconceptofprediction/distanceevaluationofthepixelsinthecurrentframe.

Second, it is not possible to establish a priori which pixels are changed ornot, so that it is not possible to nd

theparametersa n

C

withC =R ;G;B onthebasisofonepixelonly. Tosolvethisproblem, theproposed algorithm

computesaglobal estimationoftheparametersa

R ,a

G ,anda

B

usingthefollowingequations:

^ a n

R

= X

i;j2S

R R

i;j

n

R i;j

0

; ^a n

G

= X

i;j2S

G G

i;j

n

G i;j

0

; and^a n

B

= X

i;j2S

B B

i;j

n

B i;j

0

(11)

whereS

C

=f(i;j):C i;j

0

6=0gwithC=R ;G;B respectively,thehatona n

C

indicatingtheestimate. Itcanbenoted

that the estimation correspondsto the meanvalue ofthe ratiosbetween thecolorcomponents, computedall over

theimages,i.e.,foreachpixelposition. Thepredictionofthepixelvaluesisdonebythefollowingformulae:

~

R i;j

n

=^a n

R R

i;j

0

~

G i;j

n

=^a n

G G

i;j

0

~

B i;j

n

=a^ n

B B

i;j

0

(12)

ThedenitionoftheestimateofEq.11hastwoshortcomings. Firstitisquitecomputationallyexpensive,sincethree

divisions perpixel are needed. Second, sincethe meanis calculated overthe whole current(n-th) image, it takes

into consideration also those pixels that could be aected by achange that is not due to illumination. In order

to overcomethesetwoproblems, amodiedestimation strategyis elaborated. Thenewparametersare dened as

follows:

^ a n

R

= P

i;j2S R

i;j

n

P

i;j2S R

i;j

0

; ^a n

G

= P

i;j2S G

i;j

n

P

i;j2S G

i;j

0

; and^a n

B

= P

i;j2S B

i;j

n

P

i;j2S B

i;j

0

(13)

whereS=f(i;j)jpixelofposition (i;j)belongstotheimageg. Thisequationproducesestimatesthat havealways

agreatervaluethanthosefoundbyEq. 11,butthedierenceisnotsorelevant,andpracticaltestsdemonstratethat

thesenewparametersperformalmostlikethepreviousones. Moreimportant,ascanbenotedfromEq.13,onlythree

divisionsperimagehavetobedone,thusreducingthecomputationalcomplexity. Thesecondshortcomingmentioned

abovein connectionwithEq.11will besolvedbyapplyinganiterativescheme,asexplainedinthefollowingpartof

thissection.

4.2. Implementation

The change detection scheme proposed in this paper is showed in Fig.1. Inthe Parameters Estimation block the

values a^ n

R , ^a

n

G

, and ^a n

B

are computed according to Eq.13. For real time applications, where a certain speed of

calculationhastobeensured,theestimateoftheparameterscanbedoneonsubsampledimages,resultinginalower

computationalcostandstillensuringagood performanceofthealgorithm.

If the current image is aected neither by achange due to a person/objectnor by illumination variations, then

thevalues ofthese estimatesare approximately1,exhibiting some dierencesbecauseof noise,anywayproducing

predictedvaluesthatarealmostequaltothereferencevalues. Onthecontrary,ifachangeofilluminationispresent

in thescene, theestimates indicate theglobal change ofthecolorcomponentsof theimage independently foreach

R,G,andBcolorchannel.

InthePredictionblockthevaluesofthecurrentimagearepredictedfrom thoseofthereferenceimageaccordingto

theestimatedparametersofthediagonalscalingmodel. ThepredictionismadeaccordingtoEq.12. IntheDistance

blockthedistancebetweenthepredictedvaluesandtheactualonesiscomputed. Manymetricshavebeentestedin

ordertoselectonethat ensuresgoodperformance. Theonethathasbeenchosenisthefollowing:

d i;j

n

= v

u

u

t R

i;j

n

~

R i;j

n

!

2

+ G

i;j

n

~

G i;j

n

!

2

+ B

i;j

n

~

B i;j

n

!

2

(14)

whered i;j

n

indicatesthedistancebetweencorrespondingpixelsoftheactualandpredictedreferenceimageinposition

(i;j). TheformulaofEq.14doesnotactuallyrepresentatruedistanceinthecommonEuclideansense,butitallows

Figure1. Changedetectionscheme.

obtaininggoodresults in thechange detection algorithm. It isworthnoticing that thesingle termsof thesum in

Eq.14canbewrittenas:

C i;j

n

~

C i;j

n

= C

i;j

n

^ a n

C C

i;j

0

=C i;j

n 1= ^a

n

C

C i;j

0

= C

i;j

n

(C i;j

0 )

norm

(15)

whereasusualC=R ;G;B. Eq.15showsthat thedistancebetweentwoimagescanbeseenasthedistancebetween

thecurrentimageandanormalizedimage,which isobtainedbythereferenceimageimposing aglobalillumination

variation whose characteristics are similar to those of the current frame. In other words, the illuminant of the

currentimage isreported backto that ofthereferenceimage, thussolvingthecolorconstancyproblemmentioned

inSection 3.

This normalizationcanbedoneusing otherand moreaccuratemethodspropertothe colorconstancytheory, still

maintainingthe restof the scheme. This allowsto obtainabetterperformanceby just changinga singleblock of

thescheme,independentlyoftheother ones,ensuringagreaterexibilitytothewholechangedetectionscheme.

TheThresholdblockisdecidingwhetherapixelofthecurrentimagehaschangedornot. Thisdecisionismadeby

athresholdmechanismappliedto thedistance betweencorrespondingpixels.

From experimental tests it hasbeen notedthat an adaptationof thethreshold is required to obtaingood results,

especially when a strong variation of illumination occurs. These tests have shown that in presence of a positive

variationof illumination thethresholdhas to bemore selective. Forthis reason,the proposed algorithm performs

anadaptation ofthe threshold. This isdone using theparameterscomputed before, with acomputationalcost of

onedivision perimage, which isnegligible. Thethresholdisupdatedaccordingto theratioZ =a^ n

R

= ^a n

B

whichhas

beennotedtofollowtheilluminationchangeswithin acertainprecision,asitisobservedfrom Fig.2.

Thehorizontalaxisindicatestheframeindexofatestsequencefeaturingastrongvariationofilluminationnearthe

5-thframewherealampisswitchedoninthescene. OnenoticesthattheratioZ =a^ n

R

= ^a n

B

increasesafterthisevent,

andisthen stabilizingwhile thelightismaintainedon. Thisbehaviourwasveriedwith varioustestsequences, as

itwillbeshowninthenextsection.

This suggeststo use theratio Z =^a n

R

= ^a n

B

to adapt the threshold to theillumination variation. Thebinary mask

obtainedin thisblockis then:

M

1

=

1 ifd n

i;j

<

1

=Z or d n

i;j

>

2 Z

0 otherwise

(16)

where

1 and

2

aretwoparametersthat adaptthethresholdaccordingtothevideosequencecharacteristics. Itcan

benoticed from Eq.16 that the classicationconsiders either small valuesof thedistance, forwhich thethreshold

becomes more selective by a division by Z, or large values, for which the threshold becomes more selectiveby a

multiplicationbyZ.

Tohaveanideaoftheresultobtainedinthisblock,Fig.3(c)reportsthechangedetectionmaskobtainedfromEq.16.

Thereferenceand currentframe areshown in Fig.3(a)and Fig.3(b)respectively, whereit isclearlynotedthat the

0 5 10 15 20 25 0.95

1 1.05 1.1 1.15 1.2 1.25

Frame Number â _R /â _B

Figure 2. Modelingthevariationofillumination.

thealgorithm,whilethechangedueto thepersonistakenintoconsideration.

The MorphologicalFilter block performs an opening onthe binary mask M

1

. The aim of this block is twofold,

Reference Frame (a) Current Frame (b) First mask (c)

First Morph. Filter (d) AOI Selection (e) Final Mask (f)

Figure 3. Resultsofthedierentphasesofthealgorithm.

namelytoobtainabinarymaskwhichdoesnotcontainholesandwhosecontoursarequiteregular,andtoeliminate

partofthenoisepresentintherstmask. Thisoperationisverycheapfromacomputationalpointofview,sinceit

involvesoperatingonbinaryimages. Fig.3(d)reportstheresultofthisprocess.

Ascanbenoticed, aniteractiveprocessisbeingcarriedon. Theareacorrespondingtothelastmaskin Fig.3(d)is

dilatedusingasecondmorphologicallter,thusobtainingamaskwhichiscalledAOI(AreaOfInterest)asdepicted

inFig.3(e). Thismaskspeciesanareaofinterestaroundthechangedpixelsobtainedbytherstchangedetection

iteration. ThesamechangedetectionalgorithmdescribedaboveisthenappliedtothepixelslocatedwithintheAOI.

The advantageof applying this iterativeprocess lies in the fact that the estimated parameters ^a n

C

become more

accurate,sincetheycanbeprocessedinthosepartsoftheimagethathavenotchangedduetotheperson,i.e.,Eq.13

is calculatedon S =f(i;j) j (i;j) 2 (AOI) C

g, where (AOI) C

is thecomplementary set of (AOI). Furthermore,

withintheAOIthethresholdcanbelessselective,thusallowingtoobtainabetterchangedetectionmask. Thenal

changedetectionmaskisthenachievedapplying:

M

2

=

1 ifd n

i;j

<

1

=Z or d n

i;j

>

2 Z

(17)

where

1 and

2

havethesamefunction astheparameters

i

usedinEq.16. Thenalresultisreportedin Fig.3(f).

5. EXPERIMENTAL TESTSAND RESULTS

5.1. The Test Sequences

Theproposedalgorithmhasbeentestedonfourvideosequences. Eachofthemisrepresentativeofavideosurveillance

situationwherethechangedetectionalgorithmhastofacedierentproblemsinordertondavaliddetectionmask.

Thesequencesused are:

TheHall-MonitorSequence;

TheLabo1Sequence;

TheLabo2Sequence;

TheLabo3Sequence.

Theyhavebeenacquiredatarateof3framespersecond,andarecomposedof30,24,24,and24framesrespectively.

Theproposedmethodhasbeentestedonthewholesetofdata,butforsimplicityweshowonlytheresultsobtained

fromasingleframeforeachofthetestsequences. InFig.4thereferenceandcurrentframesusedtotestthechange

detectionalgorithmareshown.

The test sequences can be divided into two main groups: sequences that are not aected by a variation of the

Ref. Frame (Hall−Monitor) Ref. Frame (Labo1) Ref. Frame (Labo2) Ref. Frame (Labo3)

Curr. Frame (Hall−Monitor) Curr. Frame (Labo1) Curr. Frame (Labo2) Curr. Frame (Labo3)

Figure4. Imagesused forthetests.

illuminationof thescene, asthesequencesHall-Monitor andLabo3, andsequences thatpresentastrongvariation

oftheillumination,likeLabo1andLabo2sequences,wheretheilluminationvariationisproducedbyswitchingona

neonlampin alaboratorysetting. Thesetwotypesofsequenceswereusedin ordertotest thealgorithm bothina

relativelysimplecase,i.e., whenthere isnoilluminationchange,and inmorediÆcultsituations, which correspond

toavariationof illumination. Theproposedalgorithm shouldberobustinbothcases.

Thewell-knownHall-Monitorsequencewaschosensince,although itrepresentsaquite easysituation forachange

detectionalgorithm,itpresentssomeinterestingaspects,likethepresenceofanobjectwhichisleftinthesceneand

remainsstill (the hand-bagoftherstman that comesinto thescene), theoccurrenceof transparenciescausedby

thefact that thecolorof someparts of thesubjectswalkingin the sceneis almost thesame asthe corresponding

zonesofthebackground,etc.

The sequence Labo3 waschosen to test the performance of the iterative process which estimates the parameters

a n

introduced in the previous section. Indeed, this sequence showsin the rst frames aman walking acrossthe

scene, whosebody occupiesabig portion ofthescene. Thischaracteristiccanaect therstestimation, sincethe

parameters^a n

C

arecomputedoverthewhole image. Theseconditerationshould removethisshortcoming, andthis

sequenceisusedtoverifythisassumption.

The two remainingsequences, Labo1 andLabo2, presentbothastrong illumination variation. This variationhas

been obtained by turning on the neon lamp of the considered laboratory setting. Simultaneously, a person was

walkingthroughthescene. Theaimofthechange detectionalgorithm isthatof detectingonlythepersonandnot

thechange ofillumination. Thenature of both sequencesrendersthis task quitediÆcult, since,as canbeseen in

Fig.4,thevariationoftheilluminationcausesmanycollateralphenomena,liketheformationofshadows,reections,

anddiractionsontheobjectspresentin thescene. ForthisreasonthesesequencesarequitediÆcultto tackleand

representarelevanttaskforachangedetectionalgorithm,thusbeingagoodtestforassessingtherobustnessofthe

proposed method.

5.2. Comparisonwith other Algorithms

Inorder to test theperformanceof theproposed algorithm, itis compared to twodierentchange detection algo-

rithms. Therstalgorithmthathasbeenchosenistheoneproposedin[10],whichisbasedonthetheoryof[7]and

hasbeenbrieydescribed inSection 2. This algorithmmakesuseonly oftheluminance componentsof theimage.

Thesecondalgorithmistheoneproposedin[16]. Thisalgorithmhasbeenchosensinceitpresentssomesimilarities

withour method, althoughit isnotdesignedforchangedetectionin presence ofstrongillumination variations. In

fact itusesthecolorinformationof theimagesto handlethecolorconstancyproblem. Thisapproachissimilar to

theoneproposedinthepresentarticle.

Before presentingthe resultsof the tests, it is necessaryto notice oneimportant dierencebetween thetwo algo-

rithmsthathavebeenchosenandtheproposedone: thelatterisstructurediteratively,i.e.,toobtainthenalchange

detection mask twoscans of the images are needed, while the twoformer algorithms are single scan-based. This

fact hassomeimplicationswhencomparingthethree methods, sinceafair comparisonhastobeensured. Forthis

reason,in thefollowingpartofthissection,thenalresultofthechangedetectionalgorithmproposedinthispaper

will beshownnot onlyafter twoiterations, but also after the rstiteration. This allows afair comparisonof the

discussedmethods.

5.3. Results of the Tests

Therst testsweredevotedto theevaluation oftherobustnessof theparameterestimation donebytheproposed

algorithm. In Fig.5 the values of the ratio ^a

R

= ^a

B

are reported for the frames of the test sequences. It can be

notedthatthisratioiseectivein signalingthepresenceofanilluminationvariation. Indeed,in Fig.5(a)thisratio

maintainsalmostthesamevalues,sinceintheHall-Monitorsequencesthereisnoilluminationchange. Ontheother

hand,in Fig.5(b) andFig.5(c), which correspondto the sequences Labo1 and Labo2, itcan be clearlynoted that

theilluminationchangepresentin thesceneiswelldetectedbytheparameter^a

R

= ^a

B

. An additionalobservationis

thattherstandsecond iterationsgivealmostthesameresults,asin thesesequencesthepersons walkingthrough

thesceneoccupyasmall partofthescene.

Onthecontrary, in Fig.5(d)itis clearlyobservedthat therstestimate suersfrom thepresence oftheperson in

therstframes,butin theseconditeration,markedby( ^a

R

= ^a

B )

2

,theestimationis correct,sincetheratioremains

almost constant, as there is no change of illumination. Theestimated parameters ^a n

C

that are used to obtainthe

0 5 10 15 20 25 30

0.99 0.995 1 1.005

Frame Number (a) (â R /â

B ) 1

(â R /â B )

2

0 5 10 15 20 25

0.95 1 1.05 1.1 1.15 1.2 1.25

Frame Number (b) (â R /â

B ) 1

(â R /â B )

2

0 5 10 15 20 25

0.95 1 1.05 1.1 1.15 1.2 1.25

Frame Number (c) (â R /â

B ) 1

(â R /â B )

2

0 5 10 15 20 25

0.93 0.94 0.95 0.96 0.97 0.98 0.99 1 1.01

Frame Number (d) (â R /â

B ) 1

(â R /â B )

2

Figure5. Evaluationofparametersestimation;Hall-Monitor(a);Labo1(b);Labo2(c);Labo3(d);continuousline:

rstscan(a^=a^) ;dottedline: second scan(a^=a^) .

plotsofFig.5werecomputedbysubsamplingthetestimagesbyafactorthree bothhorizontallyandvertically.

The second tests concern the performance of the whole change detection algorithm. These tests were carried

outontheentirevideosequencesandFig.6reportstheresultsobtainedforasingleframe,wherethereferenceand

current frames usedare those of Fig.4. Theimages denoted by a

i

with i =1;2;3;4present theresult of therst

scan oftheproposedmethod, whilethose denotedby b

i

correspondto thenal detectionmask(seconditeration).

Theimagesdenoted withc

i andd

i

correspondtothedetectionmasksobtainedbythealgorithmsdescribedin [10]

andin[16] respectively.

ItcanbenoticedthatfortheHall-Monitorsequence,thethreemethodsproducesimilarresults,whileforthesequence

Labo1andLabo2theproposedalgorithmexhibitslessnoiseinthenaldetectionmask,thusoeringamorerobust

solutionto the illuminationvariations. This is obtainedmainly byits iterativestructure. Inanycase, the results

already obtainedin the rst scan are comparable with those of the other twomethods. The method proposed in

[16] presentssome diÆculties with thesequences aected by anillumination variation, precisely because it is not

designedto face strongilluminationchanges. Onthe otherhand itensuresoptimal resultsin thesequenceLabo3,

ascanbeseenin Fig.6(d

4 ).

Thealgorithmproposed in[10] exhibitsagoodnoiseremovalabilityinthesequencesLabo1andLabo2,whereasit

is lesseectivein sequence Labo3. In Fig.6(c

4

)it canbenotedthat many holesappear in the person,and this is

mainly becausethepersonpresentin thescene hasashirt ofuniform color: in thezones where thebackgroundis

also uniform theratio between theluminance components hasalowvariance, sothat the pixels corresponding to

these zones are not detectedaschanges. Thisproblem couldbe solved by loweringthe threshold, but this would

causeanincreaseofthenoisein theimage,thuscancellingtheexpectedbenet.

For our method, the threshold parameters used in the experiments were optimized for the sequence Labo1 in

ordertoobtainamaskwhichmostcloselycorrespondstotheactualchangeoftheimage. Thesameparameterswere

usedfortheothertestsequences,withoutfurtheroptimization;thisisanadvantageofthetechniquewepropose. In

fact,itshouldbenoticedthatforcomparingthealgorithms,itwasnecessarytoperformtheparameteroptimization

separatelyforeachsequence,andtheoptimalvaluesfounddiersignicantly. Theparametervaluesareprovidedin

Table1and Table2.

Method Labo1 Labo2

(Ref. frameindex=10;Cur. frameindex=61) (Ref. frameindex=10;Cur. frameindex=61)

Proposed

1

=1:3,

2

=5,

1

=1:5,

2

=2

1

=1:3,

2

=5,

1

=1:5,

2

=2

[10] T

LDD

=0:2 T

LDD

=0:3

[16]

CD

=10,

lo

= 0:5

CD

=10,

lo

= 0:5

Table 1. Parametersusedfor thesequenceswhereilluminationchangeswereproduced.

Method Hall-Monitor Labo3

(Ref. frameindex=1;Cur. frameindex=108) (Ref. frameindex=1;Cur. frameindex=16)

Proposed

1

=1:3,

2

=5,

1

=1:5,

2

=2

1

=1:3,

2

=5,

1

=1:5,

2

=2

[10] T

LDD

=0:03 T

LDD

=0:03

[16]

CD

=2,

lo

= 0:2

CD

=10,

lo

= 0:3

Table2. Parametersusedforthesequenceswhereilluminationchangeswerenotproduced.

It can be noted that, even though the number of parameters to adjust is four in the present algorithm, they

ensuregoodresultsfor sequencesthat presentverydierentcharacteristicsfrom oneto theother. Inanycasethe

researchisstillgoingonaboutreducingthenumberofthese parameters.

The morphologicaloperator that was used in the Morphological Filter block of Fig.1 is based on twoconsecutive

openingoperations,obtainedbyanerosionandadilationdonerespectivelywitha33and99squaremask. The

AOIisobtainedbyadilationoperator witha3232squaremask.

(a 1 ) (b

1 ) (c

1 ) (d

1 )

(a 2 ) (b

2 ) (c

2 ) (d

2 )

(a 3 ) (b

3 ) (c

3 ) (d

3 )

(a 4 ) (b

4 ) (c

4 )

Figure 6. Results of the change detection algorithms. (a

i

) rst scan, and (b

i

) second scan with the proposed

method; (c

i

)methodof[10];(d

i

)methodof[16];Hall-Monitor: i=1,Labo1: i=2,Labo2: i=3,Labo3: i=4.

6. CONCLUSIONS

Anewchangedetectionalgorithmwasproposedwhosecharacteristicistoberobusttostrongilluminationchanges.

Based on results known from the color constancy theory, the algorithm compensates for a possible illumination

change in animage byreportingback thecurrent illuminantto areferenceone, wherethe detectionis carriedon.

Thechangedetectionmaskwhichisobtainedisfurtherprocessedinasecondscan,inordertogetbetterresults. This

schemewasshownto bequiterobustto strongilluminationchangesevenin presenceofshadowsandhighlightings

aectingthescene,anditscomputationalcostisverylow. Furtherresearchwillbedevotedtondingabettermodel

forthechangeofcolorsinpresenceofanilluminationchange,andtoextendingthemethodtovariousrealsituations.

ACKNOWLEDGMENTS

Thisworkwasoriginally partof therstauthor'sMasterThesisperformedattheImageProcessingLaboratoryof

theUniversityofTrieste,Italy. TheresearchispresentlygoingonattheInstituteofMicrotechnology,Universityof

Neuch^atel,Switzerland,intheframeofaprojectnancedbytheSwissCommissionforTechnology andInnovation

REFERENCES

1. Atdisposalat http://www.dai.ed.ac.uk/C Vonline/LOCAL COPIES/OWENS/LECT12/node4.htm

2. B.Galvin,B.McCane,K.Novins,D.Manson,andS.Mills,\RecoveringmotionFields: AnEvaluationofEight

Optical Flow Algorithms," Ninth British Machine Vision Conf. (BMVC), Southhampton, United Kingdom,

1998.

3. E.DelechelleandJ.Lemoine,\Detectiondumouvementfondeesurunmodeled'interactionelectrique,"Vision

Interface '99,Trois-Rivieres,Canada,May1999.

4. J.MonteilandA.Beghdadi,\Unemethoderapideet robusted'estimationduchampdedeplacement," Vision

Interface '99,pp.520{527,Trois-Rivieres,Canada,May1999.

5. M. J. Blackand P. Anandan, \A framework for the robust estimation of optical ow," Fourth International

Conf.onComputer Vision,ICCV-93,pp.231{236,Berlin,Germany,May1993.

6. B.T.Phong,\Illuminationforcomputergeneratedpictures," Commun.ACM, 18,pp.311{317.

7. K.SkifstadandR.Jain,\IlluminationIndependentChangeDetectionforRealWorldImageSequences,"Com-

puterVision,Graphics, andImage Processing,46,pp.387{399,1989.

8. S.-C. Liu,C.-W. Fu, and S. Chang,\Statistical ChangeDetection withMomentsunder Time-VaryingIllumi-

nation,"IEEETrans.on Image Processing,Vol.7,No.9,pp.1258{1268,Sept.1998.

9. E.Durucan,J.SnoeckxandY.Weilenmann,\IlluminationInvariantBackgroundExtraction,"Proc.ofthe10th

International Conf.on Image Analysis andProcessing(ICIAP '99),pp.1136{1139,Venice,Italy,Sept. 27-29,

1999.

10. E.DurucanandT.Ebrahimi,\Robust andilluminationinvariantchangedetectionbasedonlineardependence

forsurveillance application,"European SignalProc. Conf., EUSIPCO 2000, Vol. II, pp.1041{1044,Tampere,

Finland,September5-8,2000.

11. T.Aach,A.Kaup,andR.Mester,\Statisticalmodel-basedchangedetectioninmovingvideo,"SignalProcessing,

31,pp.165{180,1993.

12. A. Neri, S. Colonnese, G. Russo, P. Talone, \Automatic moving object and background separation," Signal

Processing,66, pp.219{232,1998.

13. D. Toth, T. Aach, and V. Metzler, \Bayesianspatio-temporal motion detection under varying illumination,"

EuropeanSignalProcessingConf.,EUSIPCO2000, Vol.IV,pp.2081{2084,Tampere,Finland,Sept.5-8,2000.

14. L.Marcenaro,G.GeraandC.Regazzoni,\AdaptiveChangeDetectionApproachforObjectDetectioninOutdoor

ScenesUnderVariableSpeedIlluminationChanges,"EuropeanSignalProcessingConference, EUSIPCO2000,

Vol.II,pp.1025{1028,Tampere,Finland,September5-8,2000.

15. G.L.Foresti,\ObjectRecognitionandTrackingforRemoteVideoSurveillance,"IEEE Trans.on Circuitsand

Systemfor VideoTechnology,Vol.9,No.7,pp.1045{1062,October1999.

16. T.Horprasert, D. Harwood, and L.S. Davis, \A Statistical Approach forReal-time RobustBackgroundSub-

tractionandShadowDetection,"IEEEICCV'99 FRAME-RATEWorkshop,Kerkyra,Greece,Sept.1999.

17. S.A. Shafer, \Using color to separate reection components," Color Research and Applications, 10(4), pp.

210{218,1985.

18. G.D.Finlayson,M.S. Drew,andB.V.Funt,\DiagonalTransformsSuÆceforColorConstancy,"at disposalat

http://www.cs.sfu.ca/colour/publications/IC CV/i ccv abs.html

19. D. Koubaroulis,J.Matas, andJ. Kittler, \Illuminationinvariantobjectrecognitionusing theMNS method,"

EuropeanSignalProcessingConf.,EUSIPCO2000, Vol.IV,pp.2173{2176,Tampere,Finland,Sept.5-8,2000.