Local 2D Pattern Spectra as Connected Region Descriptors

HAL Id: hal-01168146

https://hal.archives-ouvertes.fr/hal-01168146

Submitted on 13 Nov 2019

HAL is a multi-disciplinary open access

archive for the deposit and dissemination of

sci-entific research documents, whether they are

pub-lished or not. The documents may come from

teaching and research institutions in France or

abroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire HAL, est

destinée au dépôt et à la diffusion de documents

scientifiques de niveau recherche, publiés ou non,

émanant des établissements d’enseignement et de

recherche français ou étrangers, des laboratoires

publics ou privés.

Local 2D Pattern Spectra as Connected Region

Descriptors

Petra Bosilj, Michael Wilkinson, Ewa Kijak, Sébastien Lefèvre

To cite this version:

Petra Bosilj, Michael Wilkinson, Ewa Kijak, Sébastien Lefèvre. Local 2D Pattern Spectra as

Con-nected Region Descriptors. International Symposium on Mathematical Morphology, 2015, Reykjavik,

Iceland. pp.182-193, �10.1007/978-3-319-18720-4_16�. �hal-01168146�

des riptors PetraBosilj

1

,Mi haelH.F.Wilkinson

2

,EwaKijak

3

,SébastienLefèvre

1⋆

1

UniversitédeBretagneSudIRISA,Vannes,Fran e

2

JohannBernoulliInstitute,UniversityofGroningen,Groningen,TheNetherlands

3

UniversitédeRennes1IRISA,Rennes,Fran e

Abstra t. We validatetheusage ofaugmented2D shape-size pattern spe tra, al ulatedonarbitrary onne tedregions.Theevaluationis per-formedonMSER regions and ompetitive performan e with SIFT de-s riptors a hieved inasimpleretrievalsystem, by ombiningthe lo al patternspe trawithnormalized entralmoments.Anadditional advan-tageoftheproposeddes riptorsistheirsize:beinglessthanhalfthesize ofSIFT,they anhandlelargerdatabasesinatime-e ientmanner.We fo usinthispaperonpresentingthe hallengesfa edwhentransitioning fromglobal patternspe tra to thelo al ones. Anexhaustive studyon theparametersandthepropertiesofthenewly onstru teddes riptoris themain ontributionoered.Wealso onsiderpossibleimprovementsto thequalityand omputatione ien yoftheproposedlo aldes riptors. Keywords: shape-size pattern spe tra, granulometries, max-tree, re-giondes riptors,CBIR

1 Introdu tion

Pattern spe tra are histogram-like stru tures originating from mathemati al morphology, ommonlyusedforimageanalysisand lassi ation[12℄,and on-taintheinformationonthedistributionofsizesandshapesofimage omponents. They anbee iently omputedusing ate hniqueknownasgranulometry[5℄ onamax-treeandmin-treehierar hy[9,19℄.

Westudy herethe appli ationof 2Dpattern spe trato ContentBased Im-age Retrieval (CBIR), to retrievedatabase images des ribing the same obje t or s ene as the query. Previous su ess in using the pattern spe tra as image des riptors omputedattheglobal[23,24℄ orpixels ale(knownasDMP [3℄or DAP [6,18℄) onvin edustoinvestigatetheirbehavioraslo aldes riptors.

StandardCBIRsystemsbasedonlo aldes riptors onsistofregiondete tion, al ulationofdes riptorsandstorageinanindex.Dierentindexings hemesare used toperformlarges aledatabasesear h[10,22℄,but allneedpowerfullo al des riptors to a hieve good performan e [21℄. To onstru t su h a des riptor,

⋆

The ollaborationbetweentheauthors was supportedbymobilitygrantsfromthe Université européennede Bretagne (UEB),Fren hGdRISISfrom CNRS,and an

keepingthegood hara teristi softheglobalversion(s ale,translationand ro-tationinvarian e,and omputatione ien y).However,toevaluatethequality andpropertiesofourproposedlo alpatternspe tra(LPS)des riptors,weneed toreexaminetheparametersusedwithglobalpatternspe traaswellasevaluate theee tofthenewparametersintrodu edbythelo al des riptors heme.

Weevaluate ourdes riptors onthe MSER regions [13℄ asthey an also be omputed on a max-tree [17℄,using the well-establishedSIFT des riptors [11℄ toobtainabaselineCBIRperforman eonadatabase.Futureworkwillin lude omparisons with SIFT extensions whi h improve performan e [1,2℄. A om-petitivepre isionisa hievedwitharotationinvariantversionofthedes riptor ombinedwithnormalized entralmoments,halfthesizeofSIFT(deeper inter-pretationoftheresultsandbesta hievedperforman e anbefoundin[4℄).

As the goal of this paper is to give an overview of hoi es and hallenges fa ed when reworking a global pattern spe trum into alo al one, we adopt a slightlyatypi alpresentationstru ture: The ba kgroundnotionsare presented in Se .2, withthefo uson howthemax-tree is usedin allparts of theCBIR system.Theexperimentalframeworkusedtotuneand evaluatethedes riptors isexplainedinSe .3.ToexaminethepropertiesoftheproposedLPSdes riptor throughtheinuen eofparametersused,themain ontribution anbefoundin Se .4,wherethedes riptorperforman eisalsopresented.Remarksonpossible improvementstothee ien yofLPS omputationaregivenin Se .5.Finally, the on lusionsaredrawnanddire tions forfuture workoeredin Se .6. 2 Ba kground

2.1 Max-tree

The on ept ofmin andmax-trees[9,19℄ ishere entralfor keypointdete tion aswellasthe al ulationoffeaturedes riptors. Were alltheirdenition using theupper andlower levelsets ofanimage, e.g.setsofimagepixels

p

withgray levelvalues

f

(p)

respe tivelyhigherandlowerthanathreshold

k

.

Givenalevel

k

ofanimage

I

,ea hlevelsetisdened as

L

k

_{= {p ∈ I|f (p) ≥}

k}

forthemax-tree,or

L

k

= {p ∈ I|f (p) ≤ k}

forthemin-tree.Their onne ted omponents(also alledthepeak omponents)

L

k,i

and

L

k,i

(

i

fromsomeindex set)arenestedandformahierar hy.Themin-treeisusuallybuiltasamax-tree oftheinvertedimage

−I

.

2.2 MSER dete tion

Peak omponents of the upperand lowerlevel sets

{L

k,i

_}

and

{L

k,i

}

oin ide with the maximal and minimal extremal regions in the ontext of Maximally StableExtremal Regions (MSER)dete torintrodu edbyMatasetal.[13℄.The dete tedregions orrespondtobrightanddarkblobs intheimageand anbe extra tedwhilebuildingthemax-treeandthemin-tree[17℄.

Extra tionofMSERreliesonthestabilityfun tion

q(L

k,i

₎

,whi hmeasures therateof growthof theregionw.r.t.the hangeof thethresholdlevel

k

.It is

fun tion orrespondtothemaximallystableregions.

We use herea simpli ation ommonly adopted by many omputer vision libraries(e.g.VLFeat[26℄):

q(L

k,i

_{) =}

|L

k

−∆,i

\L

k,i

|

|L

k,i

_|

,

(1)

where the ardinalityis denoted by

| · |

and

∆

is aparameterof the dete tor. Additional parameters ontrol the allowedregion size,limit theappearan e of toosimilarregionsandimpose alowerlimitonthestabilitys ore.

2.3 Attributesand ltering

Region hara teristi s anbe aptured byassigning themattributes measuring theinterestingaspe tsoftheregions.In reasing attributes

K(·)

givein reasing valueswhen al ulatedonanestedsequen eofregions,otherwisetheyare non-in reasing.Avalueofanin reasingattribute onatree region,

K(L

k,i

₎

,will be greaterthanthevalueofthatattribute foranyoftheregionsdes endants.

In reasingattributesareusuallyameasureofthesizeoftheregion.Wewill simplyusethearea(inpixels)oftheregion,

A(L

k,i

₎

,asthesizeattribute.Stri t shape attributesarethe nonin reasingattributesdependentonlyontheregion shape, thus invariant to s aling, rotation and translation [5℄. To indi ate the shapeofaregion,weuseanelongationmeasure alled orre tednon ompa tness:

NC

_(L

k,i

_{) = 2π}



_I(L

k,i

₎

A(L

k,i

₎

2

+

A(L

k,i

₎

6



.

(2)

I(L

k,i

₎

isherethemomentofinertiaoftheregion,andtheterm

I

(L

k,i

₎

A

(L

k,i

₎

2

without the orre tionisequaltotherstmomentinvariantofHu [8℄

I

= µ

2,0

+ µ

0,2

.

Wewillalsodire tlyusethenormalized entralmoments

n

1,1

, n

2,0

, n

0,2

, n

0,4

and

n

4,0

of the onsidered regions. These, and many moreattributes (su h as enter of mass, ovarian es,skewness orkurtosis [27℄) anbederivedbasedon rawregionmoments.

Whenthetreeisfurtherpro essedby omparingtheregionattributevalues to a threshold

t

(or using a more omplex riterion), and making a de ision to preserve orreje t a regionbased on the attribute value,we areperforming an attribute ltering. While ltering with an in reasing attribute is relatively straightforward,advan ed lteringstrategieshaveto beused whenperforming alteringwithnonin reasing(e.g.shape)attributes[5,19,24℄.

2.4 Granulometriesand globalpatternspe tra

Attributeopeningisaspe i kindofattributeltering,inwhi htheattribute usedisin reasing.Su hatransformationisanti-extensive,in reasingand idem-potent. A size granulometry an be omputed from a series of su h openings, using in reasingvaluesfor thethreshold

t

.This seriesalsosatises the absorp-tion property, sin eapplying anopening with

t

′

_{< t}

# ategories/ ategories examples sele ted u id5 31/5 allUCID ategories

with

≥

5

examples u id4 44/4 allUCID ategories

≥

4

u id3 77/3 allUCID ategories

≥

3

u id2 137/2 allUCID ategories

≥

2

u id1 262/1 allUCID ategories

image already ltered with an opening using the threshold

t

. In other words, asize agranulometry an be seenas aset of sievesof in reasing grades,ea h lettingonlydetails of ertainsizes[24℄passthrough.

Insteadoffo usingonthedetailsremaining,itisalsopossibleto onsiderthe amountofdetailremovedbetweenpairsof onse utiveopenings.Su hananalysis hasbeenintrodu edbyMaragos[12℄underthenamesizepatternspe tra.It an beseenasa1Dhistogram ontaining,forea hsize lassorlteringresidue,its Lebesgue measure (i.e. the numberof pixels in thebinary ase orthesum of graylevelsin thegrays ale ase). Su h histograms an also be omputed over dierent shape lasses, leadingto the on ept of a shape-spe tra [24℄.Finally, bothshapeandsizepatternspe tra anbe ombinedtobuildshape-sizepattern spe tra[24℄.Ashape-sizepatternspe trumisa2Dhistogram,wheretheamount ofimagedetailforthedierentsize-shape lassesarestoredindedi ated2Dbins. Previouswork[23,24℄aswellasourownexperimentssuggestthatthelower attributevalues arrymoreinformation.Thus,alogarithmi binningisusedfor bothattributes, produ ing higher resolution binsfor lowattribute values. Let

v

be theattribute valueforoneof the attributes,

N

b

thetotal desirednumber of bins and

m

the upper bound forthat attribute (whi h anbethe maximal attributevalueinthehierar hy,orasmallervalueifwede idetoignoreattribute valuesabovea ertainthreshold).Iftheminimalvaluefortheattributeis

1

(as withbothareaandthe orre tednon ompa tness),thebaseforthelogarithmi binning

b

,andthenal bin

c

,aredeterminedas:

b

=

Nb

q

_m,

(3)

c

= ⌊log

b

v⌋

(4)

Enumeratingthebinsstartingfrom

1

,the

i

-thbinhastherange

[b

i

−1

_{, b}

i

_]

. Conne ted pattern spe tra are ee tively al ulated in a single pass over a max-tree [5,24℄. Forevery region, we al ulate both the size attribute

v

1

=

A(L

k,i

₎

and shape attribute

v

2

= NC (L

k,i

₎

, and add the area of the region weightedbyits ontrastwiththeparentregion

δ

h

tothespe trumbin

S(c

1

, c

2

)

. Before using the spe trum as a des riptor, we equalize the sums in the bins as

5

Toevaluatetheretrievalperforman eoftheLPSdes riptorwithoutintrodu ing noise in the results with approximate sear h approa hes [10,22℄, we hose a relativelysmall UCID database[20℄,onwhi hwe anperformanexa tsear h. Theperforman eof ourLPSdes riptorsis omparedtoSIFT[11℄.

ThewholeUCID database ontains

1338

imagesofsize

512 × 384

pixels, di-vided into262unbalan ed ategories.After regiondete tionand des ription,a singledatabaseentryforevery ategoryis onstru ted, omprisingthe des rip-tors from all the images of that ategory. Therefore, to equalize the database entry sizes as mu h aspossible, dierent subsets of the UCID database were usedintheexperiments,wherethenumberofexamplesper ategoryis onstant forea hdatabasesubset(therequirednumberofimagesistakenfromlarger at-egories in orderprovided bythe groundtruth).Tab. 1summarizesthesubsets ofthedatabaseusedforexperimentspresentedherein.

A KD-Tree index [7℄ is then built based on the ategory des riptors, and stored for querying using the FLANN library [15℄. We then perform a query with

1

image for everydatabase ategory. The index performs a kNN sear h (

k

= 7

)withea hdes riptorofaregiondete tedonthequeryimage.Thenal ategoryisgiventhroughavotingme hanismwhereea hnearestneighbor

d

i

of aquerydes riptor

q

j

will astavoteforthe ategory

cat

(d

i

)

itbelongsto:

vote

_{(cat (d}

_i

_{)) =}

100

(L

1

(d

i

, q

j

) + 0.1) × |cat (d

i

)|

w

cat

.

(5)

L

1

(d

i

, q

j

)

referstothedistan ebetweenthesetwodes riptorsand

|cat (d

i

)|

isthe numberofdes riptorsinthe ategoryofthe

i

-thnearestneighbor.Finally,

w

cat

is aparameter ofthe experimental setup. The ve ategories with thehighest votes oresareexaminedinordertoevaluatetheperforman eofthedes riptors. The measures we used are mean average pre ision at ve (MAP5) and pre ision at one (P1). Performan e for dierent valuesof

w

cat

are shown in Fig. 1(a) and 2(d), but for all the summarized results, only the performan e fortheoptimal

w

cat

valueforea hexperimentisshown.This hoi eismadein orderto presentafair omparison,andsin enotallthedes riptorsrea htheir peak performan e for the samevalue of

w

cat

. This is additionally justied as thisparameterisnotpresentwhenusinganapproximate lassi ations heme. 4 Lo al pattern spe tra

Lo al patternspe tra (LPS)are al ulatedfrom thesele tedMSER regions.As thetwotrees ontaindierentregions,thedes riptorforamaximalMSERwill onlybebasedonthemax-tree,andsimilarlyfortheminimalMSERs.

TheLPS are al ulatedlike theglobal ones,ex ept the al ulationis done onthe orrespondingsubtree.When al ulatingthe LPSfortheMSER region

L

k,i

inthetree,weonly onsider theattribute valuesofthedes endantsofthe node.However,transitioningtothelo alversionofthedes riptorwillintrodu e

symbol signi an e value SI-LPS value SV-LPS

m

A

upperboundforarea regionsize

m

NC

upperbound fornon ompa tness

53

56

N

b

A

numberofareabins

9

10

N

b

NC

numberof non ompa tnessbins

6

M

s aleparameterfor

thesizeattribute

20000

regionsize

w(n

1,1

)

normalizedmomentweights

20

w(n

2

,0

), w(n

0

,2

)

,

10

w(n

4

,0

), w(n

0

,4

)

Toa hieveboththedesiredpropertiesand ompetitiveperforman e,the pro-posed des riptor isexplained herethroughexamining theexperiments used to establishthebestparameters.Thesummaryoftheseparameters,explained indi-viduallyhen eforth, anbefoundinTab.2.Additionally,we onsider ombining the LPS with normalized entral moments and enhan ing theperforman e by addingtheglobal patternspe tra.Theinuen eof thedatabaseontheresults isalsodis ussed.

4.1 S ale invarian e

When al ulatingaglobalpatternspe trumforanentireimage,thewholeimage size is used to determine the base of thelogarithmi binning (espe ially ifthe databaseimagesarethesamesize[23,24℄).Ifwe hoosetodeterminethebinning base for ea h region separately based on the area of that region for the lo al des riptors heme,theresultingLPSdes riptorisnots aleinvariant.

Consider two version of the same region at dierent s ales, with the area valuesbelongingto the range

[1, m

1

]

and

[1, m

2

]

respe tively. Thes ale invari-an epropertyrequiresthat,foravalue

v

1

∈ [1, m

1

]

,thebin

c

1

determinedinthe original s ale is the sameas the bin

c

2

for the value

v

2

= v

1

m

2

m

1

s aled to the range

[1, m

2

]

.However,thisisnotthe asefor

m

1

6= m

2

,as:

c

1

= log

Nb

√

m

1

v

1

6= c

2

= log

Nb

√

m

2

v

2

.

(6)

Therefore, to ensure the s ale invarian e of the des riptors, the area used to determinethebinningandthelogarithmi basehavetobethesameforallthe regions.Thisareabe omesthenaparameterofthesizeattributeinLPS, alled thes ale parameter

M

.

Usinga ommons ale

M

anbeseenasres alingalltheregionstoareferen e s ale,andhastwo onsequen es.First,foraregionofsize

m > M

,theminimal area value

v

of this region that an ontribute to the spe trum when using a ommon binning issu h that

v

′

_{= v}

M

m

= 1

, meaning that all the(sub)regions withtheareasmallerthen

m

60 %

65 %

70 %

75 %

80 %

85 %

90 %

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Precision

Category weight

ucid5

SV-LPS MAP@5

SV-LPS P@1

SIFT MAP@5

SIFT P@1

SI-LPS MAP@5

SI-LPS P@1

(a)

30 %

40 %

50 %

60 %

70 %

80 %

90 %

1

2

3

4

5

Precision

Example images per category

summary ucid5 -- ucid1

SV-LPS MAP@5

SV-LPS P@1

SIFT MAP@5

SIFT P@1

SI-LPS MAP@5

SI-LPS P@1

(b)

Fig.1:Theresults forthe nalversionof thedes riptors expressedintermsof mean averagepre isionat

5

(MAP5)andpre isionat

1

(P1)foru id5datasetforvarying ategoryweightsareshownin(a).Theresultsforu id5u id1 aresummarizedon(b) (performan eshownforoptimalweight

w

cat

foreverydataset).

withalargeenougharea anstilldisappearwhenres aling.Thisisthe asefor longthin obje tswiththewidth (alonganydimension) smallenoughto down-s aletounder

1

pixel. Su hregionsshouldbeignoredinthepatternspe trum, eveniftheirattributevaluestwiththebinning.Be auseofthis,wealso deter-minethemaximalpossiblevalueof thenon ompa tnessattribute forallofthe available areabins.

Se ond,theminimal areavalue(1 pixel)of aregionofsize

m < M

will be res aledtothevalue

v

′

₌

M

m

>

1

,and thelowerareabinsatthe ommons ale willbeempty.Therstareabin

c

min

thatwill ontaininformationisthen:

1 = b

c

min

−1

m

M

→ c

min

= log

b

M

m

+ 1.

(7)

We ompare

2

versions of thedes riptor: a) the s ale variant version (SV-LPS), where the area of ea h region is used as the s ale parameter

M

, and b) the s ale invariant version (SI-LPS) where

M

is the same for all regions. TheSV-LPSoutperformstheSI-LPSontheperformedexperiments( f.Fig.1), and mat hestheSIFT performan e. Thebest performan e fortheSI-LPS was obtained for

M

= 20000

(found experimentally) for UCID images. However, theUCIDdatabaseisnotvery hallengingintermsofs ale hange. Weexpe t theSI-LPS performan etobelessae tedthanthat ofSV-LPS whenrunning experimentsonadatabasefo usingons ale hange.

4.2 Binning parameters

With the areaattribute, the upper bound used,

m

A

, is simply the size of the region: we an plausibly expe t regions of all sizes lowerthan the size of the regionitselftobepresentinitsde omposition.

Optimal values

m

NC

for both SV-LPS and SI-LPS were determined by exam-ining the performan e of the values lose to the ones used in [23,24℄. Similar experimentswere doneto determine

N

NC

b

and

N

A

b

. Theparametertuning ex-perimentsareshownin Fig.2.

ThebestparametersfortheSI-LPSareeasytodetermine;we hose

N

NC

b

= 6

and

m

NC

= 53

. For the number of area bins, we tested both

N

A

b

= 8

and

N

_b

A

= 10

in thenal des riptor ombination(to be dis ussed in the following subse tion). Using

N

A

b

= 10

produ esbetternal resultsforSI-LPS,whi hare shown on Fig. 1. For SV-LPS, the inuen e of both parameters for non om-pa tnessis mu h slighter.Surprisingly,wefound thattheoptimalperforman e of SV-LPS rea hes an optimum at the lower value of

N

A

b

= 9

(but a higher

m

NC

= 56

thanSI-LPS). TheoptimalvaluesforbothSI-LPS andSV-LPS are listedin Tab.2.

Wealso notedthat using the optimal SI-LPS parametersin the s ale vari-antversion,we loselymat h theperforman eof ouroriginal parameter hoi e after the ombinationwith image moments. Currently, noset of parametersis performing learlybetter,but iffutureexperiments onrm thisbehavior,itis stillpreferabletouseasmaller

N

A

b

andde reasedes riptorsize.

4.3 Image momentsand globalpattern spe tra

Fiveimagemoments,

n

1,1

, n

2,0

, n

0,2

, n

0,4

and

n

4,0

,wereappendedtoanal ver-sionofallLPSdes riptors.Theweightsresultinginthebestperforman e(using the

L

1

distan e)weredeterminedbyexaminingthe ombinationoftheLPSand ea hofthemomentsseparately.Thisweightis

20

for

n

1,1

and

10

forother mo-mentsused.Additionally,anindi atorvalue

2

isaddedtoalltheLPSdes riptors originatingfromthemax-tree,and

0

forthemin-tree,thusadditionally in reas-ingthe

L

1

distan ebetweenanyminimalandmaximalMSERs.

Global pattern spe tra on their own a hieve MAP5 around

70%

on the u id5 dataset.They are added to thelist of LPS for everyimage and treated equallytootherlo al des riptors.Theinuen eof ombiningthesevalueswith SI-LPS andSV-LPSfortheoptimalparameter hoi eisshowninFig.2(d).

4.4 Regionsize and database inuen e

Before al ulatinganydes riptorsintheevaluationframeworkofMikolaj zyket al.[14℄,theregionisrstapproximatedbyanellipsewiththesame orresponding se ondmoments,andthentheregionsizeisin reasedthreetimes.Onlythenis theSIFTdes riptor al ulatedusingtheprovidedimplementation[11℄.

Sin ewewanttobeabletousethemax-treeandthemin-treeforthepattern spe tra al ulation,we hosetoworkwithan estorregionsofthedete tedMSER su hthatthesizeoftheparentisnolargerthan

xA(n

k,i

)

.Wedeterminedthat, inordertogetthesameaverageareain reaseasin[14℄,weshouldusethevalue

x

= 7.5

. The reason is that many regions have a mu h bigger parent region,

60 %

65 %

70 %

75 %

80 %

50

51

52

53

54

55

56

57

Precision

NC upper bound

noncompactness upper limit

PS-SV MAP@5

PS-SV P@1

PS-SI MAP@5

PS-SI P@1

(a)

45 %

50 %

55 %

60 %

65 %

70 %

75 %

80 %

5

6

7

8

Precision

# bins

noncompactness bins

PS-SV MAP@5

PS-SV P@1

PS-SI MAP@5

PS-SI P@1

(b)

65 %

70 %

75 %

80 %

85 %

6

7

8

9

10

11

12

Precision

# bins

area bins

PS-SV MAP@5

PS-SV P@1

PS-SI MAP@5

PS-SI P@1

( )

65 %

70 %

75 %

80 %

85 %

90 %

0.4

0.5

0.6

0.7

0.8

Precision

category weight

effect of adding moments

and indicator value

PS-SV-ALL MAP@5

PS-SV MAP@5

PS-SI-ALL MAP@5

PS-SV MAP@5

(d)

Fig.2:Parametertuningonu id5 database.Theee tofvaryingtheupperboundfor non ompa tnessis shownon (a), similarfor the amount of non ompa tnessbinson (b), and thearea binson( ).Theee t ofadding themomentsand indi atorvalue to the des riptor, with the best parameter settings is shown in (d). Note that the globaldes riptorsfortheSI-LPSare al ulatedwiththes alevalueusedfortheother des riptors,andnotusingimagesize.

Fig.1(b)summarizestheperforman eonallthesubsetsfromTab.1, allow-ing usto examinethe behaviorof thedes riptors forthein reasing size ofthe database.Theperforman eexpe tantlyde reaseswiththein reaseofdatabase sizeandde reaseofthenumberofexamplesprovidedper ategory.Asthe sepa-rateinuen eofthesetwofa tors annotbedeterminedjustfromexperiments onthesesubsets,additionaltestswere arriedoutandanalyzedin [4℄.

Besides the performan e, it is important to note here that on the largest databasesubsetused,thequeryspeedforLPSismorethan

4×

fasterthanthat forSIFT(whentheLPSdes riptorof size

60

isused).

5 Remarks on the algorithm

The system was implemented in C++. The max-tree stru ture was used for both MSER dete tion and keypoint des ription. The non-re ursive max-tree

theMSER.Themethodmethod isasfollows:

 Computethemax-treeandmin-treea ordingto[17℄.  Asthetreesarebuilt, ompute:

•

lo alminimaofthestabilityfun tion,formingthesetsofMSERregions,

•

attributevaluesforthenodesofthetrees,

•

globalpatternspe tra[24℄.

 Forea hsele tedMSERregion,repeatthe omputationofthepattern spe -tralo allyinasub-tree.

 Combinetheattributevalues,indi atorvalue

0

or

2

andthepatternspe tra toformaLPSdes riptorforaMSERregion.

 Add both global pattern spe tra[23℄ orrespondingto the whole image in the olle tionofdes riptorsfortheimage.

Unlike the al ulation of global pattern spe tra, the lo al pattern spe tra usethe onstru tedhierar hybut annotbe omputed on urrentlybe auseof dierentupperlimits(forarea)andbinnings aling value.

However,in aseofa hievingimprovedresultswiththeSI-LPS,adoptingthe s ale invariant versionto on urrent omputation anbe onsidered.While it wouldsa ri etrues aleinvarian e,ifthevalue

M

isusedasas aleparameter, and we are al ulating for aregionof size

m

, we anset the largestbin to be

[b

⌈log

b

m⌉−1

_{, b}

⌈log

b

m⌉

_]

,withthesmallestbinhavingtheupperbound

b

⌈log

b

m⌉−N

b

. Whileitisthennotalwayspossibletogetthevaluesfromthewholerangeofthe largestbin, thebin valuesof the hildren anbe usedby their parents.When the upper bound of the largest bin hanges, the hild values an still be used withdis ardingthevaluesfromthesmallestbin:thes aleofthosedetailsistoo lowtobe onsidered.

6 Dis ussion and on lusion

After su essfullyapplying globalpattern spe train CBIR ontext [23,25℄, we nowattemptto onstru talo alregiondes riptorbasedonthepatternspe tra. On the hosen subsets of the UCID database [20℄, the results obtained were betterthanwhenonlyusingglobalpatternspe tra(almost

20%

in MAP5on u id5),andmat hedtheperforman eoftheSIFTdes riptor.

Theproposed LPS des riptors haveanother advantage.In addition to the des ription al ulationpro essbeingslightlyfasterforthepatternspe trathan for the SIFT des riptors, our des riptors length is only

47%

of the length of SIFT. Thismakesusingthesedes riptorsmu hfasterperforming

262

queries onanindexofthesize

262

(u id1 dataset)took over

4

timeslongerusingSIFT des riptors.Thissuggeststhat(espe iallyin larges aleCBIRsystems), we an usemoreexampleimagesinordertoenhan ethepre ision,whilestillperforming fasterthanSIFT.

auses a de rease in performan e. We plan to evaluate both the SI-LPS and SV-LPS onadatabasefo usedons ale hangesto determinethevalueoftrue s aleinvarian einsu h ases.

Despitetheparametersandthedes riptor invarian ewhi h havetobe fur-ther studied, mat hing theSIFT performan eon thethree subsets ofthe u id datasetwithades riptoroflessthanhalfthelengthofSIFTisverypromising. Additional su essful experiments were performed and analyzed in [4℄. It also prompts forevaluating theLPS performan e withlarge s aleCBIRsystem. It is probable that the results ould beeven further improved by ombining the urrentLPSwithpatternspe trabasedonothershapeattributes,likein[23℄.

Lastly,the

L

1

distan e, designedto ompareve torsofs alarvalues,isnot the best hoi e for omparing histogram-like stru tures. Using dierent dis-tan es, or even divergen es (e.g. [16℄) whi h take into a ount the nature of thedes riptorshouldalsoimprovetheperforman e.

Referen es

1. Arandjelovi¢, R., Zisserman,A.: Three things everyone should know to improve obje tretrieval.In:ComputerVisionandPatternRe ognition(CVPR),2012IEEE Conferen eon.pp.29112918. IEEE(2012)

2. Bay,H.,Ess,A.,Tuytelaars,T.,VanGool,L.:Speeded-uprobustfeatures(SURF). Computervisionandimageunderstanding110(3),346359(2008)

3. Benediktsson,J.A.,Pesaresi,M.,Arnason,K.: Classi ationand Feature Extra -tionforRemoteSensingImagesfromUrbanAreasbasedonMorphologi al Trans-formations. IEEE Transa tionson Geos ien e and Remote Sensing 41(9), 1940 1949 (2003)

4. Bosilj, P.,Kijak,E., Wilkinson,M.H.F., Lefèvre,S.: Shortlo aldes riptors from 2D onne tedpatternspe tra,submittedtoICIP2015

1

5. Breen, E.J.,Jones, R.: Attributeopenings, thinnings, and granulometries. Com-puterVisionandImageUnderstanding64(3),377389(1996)

6. Dalla Mura,K., Benediktsson, J.A.,Waske, B.,Bruzzone,L.: Morphologi al At-tributeProlesfor theAnalysisof VeryHighResolutionImages. IEEE Transa -tionsonGeos ien eandRemoteSensing48(10),37473762(2010)

7. Friedman,J.H.,Bentley,J.L.,Finkel,R.A.:Analgorithmforndingbestmat hes in logarithmi expe ted time. ACM Transa tions on Mathemati al Software (TOMS)3(3),209226(1977)

8. Hu,M.K.:Visualpatternre ognitionbymomentinvariants.InformationTheory, IRETransa tionson8(2),179187(1962)

9. Jones,R.:Componenttreesforimagelteringandsegmentation.In:IEEE Work-shoponNonlinear Signaland ImagePro essing, E.Coyle,Ed., Ma kina Island (1997)

10. Lejsek,H.,Jónsson,B.T.,Amsaleg,L.:NV-Tree:NearestNeighborsattheBillion S ale. In: Pro eedings of the 1st ACM International Conferen e onMultimedia Retrieval.pp.54:154:8.ICMR'11(2011)

tionaljournal of omputervision60(2),91110(2004)

12. Maragos,P.:Patternspe trumandmultis aleshaperepresentation.Pattern Anal-ysisandMa hineIntelligen e,IEEETransa tionson11(7),701716(1989) 13. Matas, J., Chum, O., Urban, M., Pajdla, T.: Robust wide-baseline stereo from

maximally stableextremalregions. Imageand vision omputing22(10),761767 (2004)

14. Mikolaj zyk,K.,S hmid,C.:Aperforman eevaluationoflo aldes riptors.Pattern AnalysisandMa hineIntelligen e,IEEETransa tionson27(10),16151630(2005) 15. Muja, M.,Lowe,D.G.:FastApproximateNearestNeighborswithAutomati Al-gorithmConguration.In:InternationalConferen e onComputerVision Theory andAppli ationVISSAPP'09).pp.331340.INSTICCPress(2009)

16. Mwebaze, E., S hneider, P.,S hleif, F.M., Aduwo,J.R., Quinn,J.A., Haase, S., Villmann,T., Biehl,M.:Divergen e-based lassi ationinlearningve tor quanti-zation.Neuro omputing74(9),14291435(2011)

17. Nistér,D.,Stewénius,H.:Lineartimemaximallystableextremalregions.In: Com-puterVisionECCV 2008,pp.183196.Springer(2008)

18. Ouzounis,G.K.,Pesaresi,M.,Soille,P.:DierentialAreaProles: De omposition PropertiesandE ientComputation.IEEETransa tionsonPatternAnalysisand Ma hineIntelligen e34(8),15331548(2012)

19. Salembier, P., Oliveras, A., Garrido, L.: Antiextensive onne ted operators for image and sequen e pro essing. Image Pro essing, IEEE Transa tions on 7(4), 555570(1998)

20. S haefer,G.,Sti h,M.:UCID:AnUn ompressedColourImageDatabase.In: Ele -troni Imaging2004.pp.472480.InternationalSo ietyfor Opti sandPhotoni s (2003)

21. S hmid,C., Mohr, R.: Obje tre ognition usinglo al hara terization and semi-lo al onstraints.IEEETransa tionsonPatternAnalysisandMa hineIntelligen e 19(5),530534(1997)

22. Sivi ,J.,Zisserman,A.:VideoGoogle:E ientvisualsear hofvideos.In:Pon e, J., Hebert,M., S hmid,C., Zisserman,A. (eds.) Toward Category-Level Obje t Re ognition,LNCS,vol. 4170,pp.127144.Springer(2006)

23. Tushabe,F.,Wilkinson,M.H.F.:Content-basedimageretrievalusing ombined2D attributepatternspe tra.In:Advan esinMultilingualandMultimodal Informa-tionRetrieval,pp.554561.Springer(2008)

24. Urba h,E.R.,Roerdink,J.B.T.M.,Wilkinson,M.H.F.:Conne tedshape-size pat-tern spe tra for rotation and s ale-invariant lassi ation of gray-s ale images. PatternAnalysisandMa hineIntelligen e,IEEE Transa tionson29(2),272285 (2007)

25. Urba h,E.R.,Wilkinson,M.H.F.:Shape-onlygranulometriesandgrey-s aleshape lters. In: Pro . Int. Symp. Math. Morphology (ISMM).vol. 2002, pp. 305314 (2002)

26. Vedaldi,A.,Fulkerson, B.:VLFeat: AnOpenandPortable LibraryofComputer VisionAlgorithms.http://www.vlfeat.org/ (2008)

27. Wilkinson,M.H.F.:Generalizedpatternspe trasensitivetospatialinformation.In: PatternRe ognition, InternationalConferen eon. vol.1,pp.1002110021. IEEE ComputerSo iety (2002)