A representation of contextual relationships knowledge in images

(1)

lamsade

LAMSADE

Laboratoire d ’Analyses et Modélisation de Systèmes pour

l ’Aide à la Décision

UMR 7243

Mai 2012

A representation of contextual relationships

knowledge in images

N. V. Hoang, V.Gouet-Brunet, M.Rukoz

CAHIER DU

(2)

A representation of ontextual relationships knowledge in images Nguyen Vu Hoang

1,3

, ValérieGouet-Brunet

2

,Marta Rukoz

1,3

1:LAMSADE-UniversitéParis-Dauphine-Pla edeLattredeTassigny-F75775ParisCedex16

2:CEDRIC/CNAM-292,rueSaint-Martin-F75141ParisCedex03

3:UniversitéParisOuestNanterreLaDéfense-200,avenuedelaRépublique-F92001NanterreCedex

nguyenvu.hoangdauphine.fr, valerie.gouet nam.fr,marta.rukozdauphine.fr

23 mars 2012

Abstra t

Thisreportisfo usedonthestudyofmethodsforimageretrievalin olle tionofheterogeneous

ontents.Thespatialrelationshipsbetweenentitiesinanimageallowto reatetheglobaldes ription

oftheimagethatwe alltheimage ontext.Takingintoa ountthe ontextualspatialrelationships

inthesimilaritysear hofimages anallowimprovingtheretrievalqualitybylimitingfalsealarms.

Wedenedthe ontextofimageasthepresen eofentity ategoriesandtheirspatialrelationships

intheimage.

By studying statisti ally the relationships between dierent entity ategories on LabelMe, a

symboli images databasesof heterogeneous ontent, we reate a artographyoftheir spatial

re-lationships that an be integrated in a graph-based model of the ontextual relationships, the

prin ipal ontribution of this report. This graph des ribesthe general knowledge of everyentity

ategories.Spatialreasoning onthis knowledge graph anhelp improving tasks ofimage

pro es-singsu hasdete tionandlo alization ofanentity ategorybyusingthepresen eofanotherone.

Further,thismodel anbeappliedtorepresentthe ontextofanimage. Thesimilaritysear h

ba-sedon ontext anbea hievedby omparing thegraphs,then, ontextualsimilaritybetweentwo

imagesisevaluatedbythesimilaritybetweentheirgraphs.Thisworkwasevaluatedonthe

symbo-li imagedatabaseofLabelMe.Theexperimentsshoweditsrelevan eforimageretrievalby ontext.

Keywords :Image,similaritysear h,spatial relationships,image ontext.

1 Introdu tion

Theinterpretationofimagesbyama hinerequirestohavearepresentationofimagesprepro essed

manuallyor automati ally. Thisrepresentation an be builtfrom visualfeatures(su has olor,shape

of elements in images) or higher level information (su h asspatial relationships between elements or

(3)

say that what makes su h humans' possible aptitudes is their ability to interpret visual features of

images by using prior knowledge. Prior knowledge is very often related to the presen e of multiple

entities inanimage and to thespatial information linkingthem,that an be alledthe ontext ofthe

image.Humans an in orporate this knowledge to analysetheimage and to reate personal semanti

on epts.A ording to [8 ℄,image interpretation an be lassied into threelevels of omplexity:

Level 1 : interpretation is based mainly on primary featuressu h as olor, texture, shape,

seg-mentedregions, interestpointsand/orthespatiallo ation ofimageelements. Thesefeaturesare

rather obje tive and their estimation is performed dire tly, it does not require any knowledge

base. Many approa hes of image retrieval or of image ategorization an be lassied into this

level (e.g.BagofFeaturesandall its derived approa hes).

Level 2 : interpretation involves some degree of logi al inferen e on erning the image ontent.

At this level, queries are performed in order to retrieve entities of a given type or to retrieve a

spe i entityfromanother.Theneed ofapro essedknowledgebaseisobvious. Thisknowledge

base an ontain low-level information of entity ategories (e.g olor, shape), relationships

bet-ween ategories (e.g. orrelations, onditional probabilities, spatial relationships), andmore.

Level 3 : interpretation is based on symboli features. At this level, a signi ant amount of

high-level reasoningaboutthemeaning and purposeon theentities or on ba kground of images

an be involved. The result of this pro essing isthat an image an be linked to a on ept bya

subje tivejudgement.Todate,humansareonlyoneswho anproposeanee tive interpretation

ofan image at thislevel.

Mostof theapproa hesproposedlie between levels 1and 2.Itisstill di ultto lie between levels

2 and 3 that refer to high-level semanti image retrieval [12℄. The main eort is to onne t low-level

featuresto high-levelsemanti s ofimages. The ee tive approa hesto dateare:

1. usingma hinelearningmethods inorderto asso iate high-level on epts to low-levelfeatures,

2. takinginto a ount userfeedba kinorderto improve subje tive on epts,

3. inferringvisual ontent basedon textualinformation extra ted from image ontext,

4. usingentity ontology inorderto dene high-level on epts.

Most ontentbasedimageretrievalsystemsexploit a ombination oftwoormoreofthesemethods

in order to perform high-level semanti image retrieval (see [24, 8, 18 , 27, 6 ℄). Although the results

obtained are promising, designing systems that really understand image ontent at semanti level is

stilla openproblem.

In this paper, we propose a model to represent a general knowledge about relationships su h as

spatialones between entity ategories existing inanimage database.Furthermore,this model an be

used to des ribe the ontext of a given image. The denition of image ontext is dis ussed in the

se tion2.2. Finally,we observe thatanimage maybelinked to multiplesubje tive intera tions,then,

we hope to attribute a semanti meaning to ea h ontext image that an fa ilitate image retrievalor

(4)

and onsider thatthese on epts areknown.

Inthese tion2,wepresentthedenitionofimage ontextandseveralspatialrelationshipsbetween

ategories. The se tion 3 presents the on epts and denitions of our graph model. In the next, we

dis usstheevolution apa ityof the graphto the newknowledge and spatialreasoning inthese tion

4and 5.Finally, inthe se tion6,we present several experiments to evaluateour graph model.

2 Initial denitions

In this se tion, we present several denitions like spatial relationships and image ontext before

presenting our prin ipal work.

2.1 Spatial relationship

In our framework, we are rstly interested in the representation of spatial relationships between

symboli obje ts inimages, alled entities. In CBIR,embedding su h information into image ontent

des ription provides a better representation of the ontent as well as new s enarios of interrogation.

Thespatial relationships an be the unary, binary,andternary relationships.

We all unary relationship, the relationship between an entity and its lo alization in an image,

where lo alization is dened as a region or an areaof the image. Areas of an image an be

represen-ted in dierent ways like quad-tree or quin-tree, see for example [20, 28 ℄. Sin e we do not have any

knowledgeaprioriofthelo ationofthe ategoriesintheimages, weproposetosplitimagesinaxed

numberof regularareas (i.e. equal size areas). First, we divideea h image ina xedsized grid. Ea h

ell of this grid, alledatomi area, is represented bya ode. Fig.1and 2 depi t a splitting in

9

or in

16

dierent basi areas and theirs odes, respe tively. We then ombine these odesto present more omplex areas,byexample for 9-area splitting, ode

009

represents area( ) grouping together areas

001

( ) and

008

( ).

001 008 064

002 016 128

004 032 256

Figure 1 Codesinunaryrelationshipbysplittinganimageinnine areas.

00001 00016 00256 04096

00002 00032 00512 08192

00004 00064 01024 16384

00008 00128 02048 32768

Figure 2 Codesinunaryrelationshipbysplittinganimagein 16areas.

(5)

lassied as topologi al, dire tional or distan e-based approa hes (see [11 ℄ for more details), and an

be applied on symboli obje ts or low levelfeatures. Here, we have fo ussed on relationships between

theentities of the database des ribed intermsof dire tional relationships withapproa h 9DSpa [17℄,

oftopologi al relationships [9, 10℄and of a ombination of them with2Dproje tions[19 ℄. We do not

useorthogonal[3℄ and9DLTrelationship [2℄be ause ofits in onvenien es mentioned in[17 ℄.

A ternary relationship des ribesa relationship of a triplet of ategories. To our knowledge, a few

approa heswere proposedto des ribe risp triangular relationships of threesymboli entities.We an

mentionTSRapproa h[13℄andourapproa h

∆

-TSR(see[16℄). Byapplyingtoasetofheterogeneous symboli entities that do not have xed shape and size, these approa hes annot des ribed fully

tri-angularspatial relationships between symboli entities sin e theytakeinto a ount onlythe enterof

ea hentityasrepresentation of it.

2.2 Image ontext

Inanimage, there ognition or dete tionofentity ategoryrequiresdierent information fromthe

raw image data. A ording to [26℄, in the real world, there exists a strong relationship between the

environmentsand entities foundwithinitor between the entities.Entitiesarenever inisolation.They

an tend to o-vary with others entities and parti ular environments for providing a ri h olle tion

of ontextual asso iations.The re ognition or dete tion will be a urate and qui k ifentities usually

appear ina familiarba kground.Then, initially,we andene that the ontext ofan image des ribes

allpossible typesof relationshipbetween the entities in thisimage, or between theentities and

ba k-ground of this image. The use of image ontext an bring a strong interest not only for re ognizing

or dete ting the entity ategory but also for image retrieval. For the re ognition or dete tion of an

entity ategory, it is evident to examine the general ontext of image if thelo al features are

insu- ient (e.g.entityis small,orappearspartially). For image retrieval, the omparisonof image ontexts

anhelptolteroutthefalsealarmsbeforeenterinthestepof omparisonofvisual ontentsofimages.

By usingthevisual featuresinimage,the ontext an bedes ribed byrelationships between lo al

informations and global information of the image. This ontext denition an drive to a hard work

of image pro essing. Another natural way of representing the ontext of an image is using the

o-o urren e relationships ofits entities. In thereal world, the o-o urren e might happen at a global

level, for example a bed room will predi t a bed, or at a lo al level, for example a table will predi t

thepresen eofa hair. Aprobabilisti problem an bealso asso iated inthis ase.More omplex,the

spatialrelationships between entity ategories inimages an be taken intoa ount.Ingeneral, thatis

di ult to have an exa t denition of ontext; ea h ase of use an depend on a parti ular ontext

denition.

Here, we try to study dierent relationship that ould be present in images. A ording to [14℄,

entities inanimage anbethings (e.g. ar,people) or stuff(e.g.road,buildings,more pre isethat

(6)

Stu-Thing:textureregionsthat allows to predi tthepresent ofan entity ategory.

Stu-Stu:relationships between regionsof images.

S ene-Thing : s ene information su h as s ale, global dire tion that allows to determine the

lo ationof anentity ategory.

S ene-Stu:s eneinformationsu hass ale,globaldire tionthatallowstodeterminethelo ation

ofa region.

In our framework, we do not dierentiate the entities present in image as "Thing" or as "Stu"

be ause we are interested in symboli obje ts that are represented by polygons. These entities are

lassiedsimplyby ategory.Wedenetheimage ontext bythepresen eofentity ategoriesinimage

and by thespatial relationships between these entity ategories. The presen e of at least an instan e

ofanentity ategorywill onrmthepresen eofthisone.Thespatialrelationships between ategories

inimagewillberepresentedinageneral way(e.g.probabilities).There aretwoprin ipalwaysofusing

the ontext inavision system:

Apriori:inthis way,the ontext servesto lo atetheentities,to limit thesear hingregion,and

tode rease the retrievaltime(for examplethe approa hesproposedby[26 , 14,25℄).

Aposteriori :the ontext servesto obje tre ognition ifthelo alinformation isnot su ient, it

anhelpto redu e theambiguitiesofthepresentsof obje tsinthesame s ene(forexample the

approa hesproposedby[23, 22,5 ℄).

There has been a growing interest in exploiting ontextual information for image retrieval,

las-si ation or obje t dete tion, re ognition. Dierent te hniques have been exploited to des ribe the

ontext of image for this purpose. Intuitively, the spatial lo ations of obje ts and ba kground s ene

from global view an be used as inside-image ontext. Further, the ombination of obje t dete tion

and lassi ationtaskstogether anprovidenatural omprehensive ontextforea hotherwithoutany

external assistan e. Moreover, a knowledge database, onsidered asa external element, based on

ma- hinelearningSVMorprobabilisti te hnique,allowstoenhan eothertasksaslo alization,dete tion,

et . Ourapproa h proposedinthenext se tions isa priorione.

3 A Graph-based Knowledge Representation

In this se tion, we present how to represent a knowledge between entity ategories in images by

usingagraph.The on eptsanddenitionsofthisgrapharepresentedinse tion3.1.Toavoidbuilding

anunreadable graph,theattributes ofnodeandthegraph onstraintsaredis ussedinse tion3.2and

3.3respe tively.Finally,se tion 3.4presentsabrief example oftheuseof thegraph.

3.1 Con ept and denitions

In reality, events an be expressed by two notions : entity and relationship. For example, if we

have anevent "ourlaboratory invited aprofessor last month", then this event an be representedby

two entities : "our laboratory" and "a professor" that have a relationship "invite" of attribute "last

month". We knowthat a general knowledge presents a general event based on on rete eventswhi h

(7)

maybe linked bya relationship of type "invite". Based on this argument, we wouldliketo represent

the learnt knowledge by using a graph on ept developed only on two notions of " ategory" and of

"relationship". In this graph, the instan e of ategory (an entity) is not meaningful to guarantee a

generalrepresentationof aknowledge. However, entities arealwaysexamined beforebuildingagraph.

Thereason wasexplained previously :ageneral event is basedon parti ularsevents.

Normally,in a lassi graph, a vertex (ora node) represents a ategory and an edgerepresents a

relationship. We observed thata "relationship" maybe unary,binary,ternary or n-nary relationship,

then a "relationship" an on ern one or many ategories. A lassi graph an represent only the

binaryrelationships between two verti es. By using a hypergraph, a generalization of a graph [4℄,an

edge an onne t any number of verti es (see Fig.3(a)). However, this onne tion of a set of verti es

is represented by only an edge. We know that, between two or more " ategories", there are many

"relationships".It means thatdierent edges an have thesame end nodes, thenamultigraph [1℄ an

beanotheralternativegraph.However,amultigraphallowstodes ribemultiplerelations betweentwo

andonlytwoverti es(seeFig.4(a)). Forour on ept,wewouldlike tousetheadvantages ofthesetwo

graph models, and we know that a bipartite graph [29 ℄ an model the more general multigraph and

hypergraph (see examplesofrepresentation ofhypergraph inFig.3(b) andof multigraphinFig.4(b)).

(a)Hypergraph (b)Bipartite graph

Figure 3 An example ofhypergraph andits bipartite representation.

(a)Multigraph (b)Bipartite graph

Figure 4 Anexample of multigraphand its bipartite representation.

We would like to present multiple relations between multiple verti es, that is why we de ided to

(8)

ofnodes:a ategorynode,denoted

C

andarelationshipnode,denoted

R

.Wegiveadenitiontoea h typeof node inour graph

G

:

A ategory node

C

represents the existen eof aset of ategories ina same environment, i.e. in thesame database.For aset of ategories

K = {cat

i

}

,its representation node is

C

{cat

i

}

or

C

, we denote this node

R

type

J

. From a

R

type

J

, we an learn all possible ongurations of relationship

type

involved from

J

. In our framework, we are espe ially interested in spatial relationships,for example,relationship

type

anbethetopologi al spatialrelationship [10 ℄ that des ribesdierent ongurations :disjoint,joint, overlaps,insides ,et .

Now, graph

G

is dened as:

G = (V, E)

(1)

Knowing that

V

is the setof nodesinthegraph :

V

= {C

K

} ∪ {R

J

}

(2)

Thisgraphis anundire ted graph.Then,

E

istheset ofedges :

E =

{e

(C

K

,R

J

)

|∀C

K

, R

J

∈ V ∧ K ⊆ J}

∀C

K

, R

J

∈ V ; e

(C

K

,R

J

)

⇒ e

(R

J

,C

K

)

∀C

K

, C

J

∈ V ; ∄e

(C

K

,C

J

)

∀R

K

, R

J

∈ V ; ∄e

(R

K

,R

J

)

(3)

Figure5 givesanexample of graph(at3 levels).

3.2 Attributes of a node

Ontheotherhand,our requirementisthatgraph-basedrepresentationmustbesimpleto ompute,

leartounderstand,andextendibletorepresentanew omplexgeneralknowledge.Fromthisquestion,

we dene spe i attributes asso iated to ea h node, ategory or relationship, in se tions 3.2.1 and

3.2.2.

3.2.1 Level attribute

To avoid building an unreadable graph, based on the idea that say that a omplex knowledge is

developedfromasetofbasi ones,wesplitourgraphintomanydierentlevels.Agraphlevelindi ates

(9)

level0,1,and 2respe tively.

is omposed of a set of

C

and

R

nodesthat have the same

|cat

i

|

. Thus, we an onsider thelevel as an attribute of the node, denoted

lev

,that is dened as :

lev = |cat

i

| − 1

. Our idea is that a higher levelnode an be built from lower level nodes. A low level node an onne t only to one higher level

byedges between

C

nodesoflower leveland

R

nodesof higherlevel. In onsequen e, we redeneset

E

:

E = {e

_(C

_K

,R

J

)

|∀C

K

, R

J

∈ V ∧ K ⊆ J ∧ (C.lev = R.lev ∨ C.lev + 1 = R.lev)}

(4) Note thatwe an expand thenumber oflevels inthe graph aswe need. Ifwe study

N

ategories, then at the level

l

, we an have in maximum

N!

(N −l)!

nodes

C

. A high number of levels an in rease onsiderably thenumber ofnodesin thegraph, however, inreality, we an nd many ategories that

never o ur together. For example, in [15℄, by studying

86

dierent entity ategories in a dataset of heterogeneous ontent,we found

879

ouplesof ategoriesthatnevero urtogether amongaset

7310

ofpossible oupleand only

38031

present triplets intotalamong

102340

possible triplets.

In Fig.5, we show the on ept for a 3-level graph. Con retely, we an model unary, binary, and

(10)

We know thata knowledge an be eithertrue, or false, or un ertain. In fa t, for example,we are

not ertain to onrm the existen e of aliens be ause of the limit of our s ienti knowledge. "The

earth isa square" is a false denition. In addition, a true onrmation ould be ome a false one;for

example, nowadays the onrmation "the earth is a enter of universe" is not true. To omplete the

graph on eption, weproposeto addanattributeonsu h statusfor ea hnode ofgraph,denoted

stat

. A node in graph an be either true, or false, or un ertain. These statuses an be modied to

deal with a new situation ifne essary.Figs. 5, 8,and 9 show examples of true nodes. To represent

a un ertain node, we use a dis ontinuous line node as shown in Fig.6(a). Finally a false node is

represented by a strike-through node as shown in Fig.6(b). Note that false and true statuses are

onsideredas onrmations,though,un ertainstatuswillbeusedasindu tionthatisnotveriedyet.

(a) Un ertainnoderepresentation (b)False node representation

Figure6 Two additional statusesof anode.

A

C

J

butfor dierent ongurations.We denotethesetof ongurationsin

R

type

J

:

R

type

J

.Φ

. Let

type.Φ

betheset ofall possible ongurations ofthe relationship

type

.In onsequen e :

R

type

_J|stat=true

.Φ ∪ R

type

_{J|stat=uncertain}

.Φ ∪ R

type

_{J|stat=f alse}

.Φ ⊆ type.Φ

(5) And:

R

type

_J|stat=true

.Φ ∩ R

type

_{J|stat=uncertain}

.Φ ∩ R

type

_{J|stat=f alse}

.Φ = ∅

(6)

Note that a onguration of relationship

type

belongs to only one

R

node among three dierent status

R

nodes of

J

and

type

(seeFig.7 foran example).

By addinga statusto anode,there aretwopossible s enariosto represent aknowledge base:

We begin our knowledge representation with a graph ontaining all possible ombinations on

ategories and on relationships of universe (i.e. there is no limitation on the number of entity

ategories and of types of relationship). It is evident that these nodes are un ertain initially.

(11)

Figure 7 Three dierent statuses of

R

nodesowning thesame

type

and set

J

.

thesenodes.But thiss enario is omplex and willnot be taken into a ount inour framework.

A knowledge graph is built dire tly from a given database. An initial graph ontains only the

true or false nodes. The number of ategories and the types of relationship are limited. This

graphevolute byadding newknowledge. Our workinthefollowing will respe t thiss enario.

3.3 Graph onstraints

Hereanother hallengeisthatthe omparison oftwo graphsmaynotinvolvea ostly omputation.

To avoid to ompli ate the graph that an impa t on graph omputation su h as omparison, some

onstraints mustbe asso iated to it, onnodes(se tion3.3.1), onstatuses (se tion3.3.2)and onedges

(se tion3.3.3).

3.3.1 Node onstraints

Thepresen e ofaset ofentity ategories isrepresentedbyone

C

node only.Thus, a

C

node must be unique. It an be identied with a unique identi ation, denoted

id

and omputed by fun tion of equation 7 :

C

K

.id = F

ID

C

(K)

. Note that a

cat

i

an be identied by a unique

id

(that is an integer) andthat

N

Categ

isthe total numberof ategories examinedinsystem.The

id

attribute ofthe

C

node is an integer omputed from the

id

of

cat

i

|cat

i

∈ K

. It will allow to lo ate qui kly a

C

node in the graphby usingahash fun tion.

(12)

F

ID

C

(K) =

P

|K|

i=1

cat

i

.id ∗ (N

Categ

)

(i−1)

∀cat

i

, cat

j

∈ K; i < j ⇒ cat

i

.id < cat

j

.id

(7)

In onsequen e, we obtain:

∀C

K

∈ V : ∄C

J

∈ V |C

K

.id = C

J

.id

.

Similarly,a

ID

R

(K) =

type + toString(stat) + toString(F

C

ID

(K))

(8)

where

toString(s)

allows to transform a data type (boolean, number, et .) to a string. Then the

id

of a

R

node isa stringthatisunique. Thus, we obtain:

∀R

J

, R

K

∈ V : (J 6= K) ∨ (J = K ∧ R

J

.type 6= R

K

.type)

∨(J = K ∧ R

J

.type = R

K

.type ∧ R

K

.stat 6= R

J

.stat)

(9)

It isevident that, for a type of relationship, there maybe manydierent

R

nodes. It meansthere aremanyinstan esof

R

l

(seeFig.5).

3.3.2 Status onstraints

To be able to ompare the three statuses, we impose that true>un ertain>false. Finally, to

avoid absurdrepresentations inthegraph,wepropose three onstraints on erning

stat

denition: Therst onstraint on erns the

C

and

R

nodesthatare onne ted and at asame level:

∀C

K

, R

J

∈ V : (I = J ⇒ C

K

.stat >= R

J

.stat)

(10) Notethat:

I = J

an berepla edby(

∃e

(C

K

,R

J

)

∈ E ∧ C

K

.lev = R

J

.lev

).

The se ond onstraint on erns the lower level

C

node and the higher level

R

node that are onne ted:

(13)

Herenotethat:

I ⊂ J

anbe repla edby(

∃e

(C

K

,R

J

)

∈ E ∧ C

K

.lev = R

j

.lev − 1

). Thethird onstraint on erns two

C

nodesoftwo dierent levels :

∀C

K

, C

J

∈ V : I ⊂ J ⇒ C

K

.stat >= C

J

.stat

(12) We an observe thatthe status of a ategory node plays an important role.It is ertain that the

statusofa

R

nodedependson thestatusofa

C

node.We annot denythat thenon-existen e ofa

C

node will reje t all

R

nodes linked with it. For example, we an say that there are many resear hes on erningrelationshipbetween alien ategoryandearth ategory;andoneday,ifwe onrm that

this alien ategory do not exist (this ategory has a false status), then all asso iated relationships

be ome false. Furthermore, it is also evident that the status of a higher level node depends on the

statusofalowerlevelnode.Forexample,ifwe onrmthatinanenvironment,thereisno

A

ategory (

C

{A}

.stat

is false), then it is impossible for a ouple of ategories

(A, B)

to be present in this environment (thus,

C

{A,B}

.stat

be omesfalse too).

3.3.3 Edge onstraints

We imposetwo onstraints fortheedges between

C

nodesand

R

nodes:

The rst onstraint is for two nodes of same level. Node

C

K

an be linked to node

R

J

at the samelevelifand only ifthey areassigned tothe sameset of ategories, itmeans too

K = J

:

∀R

J

∈ V : ∃C

K

∈ V |C

K

.lev = R

J

.lev ∧ K = J ∧ ∃e

(C

K

,R

J

)

∈ E

(13) The se ond onstraint is for two nodes of dierent levels. A node

C

K

of a lower level an be

linkedto a node

R

J

ofa next higherlevelifand only if

K ⊂ J

:

∀R

J

, C

K

∈ V |(R

J

.lev = C

K

.lev + 1) ∧ (K ⊂ J) : ∃e

(C

K

,R

J

)

∈ E

(14) Thislast onstraint onrmsthefa tthatonelowerlevelis onne tedtoonlyonenexthigherlevel.

3.4 Examples

Tomake learthemodelofourgraph,weproposetwoexamplesinFig.8and9.Intherstexample,

we des ribe spatial binary relationship, represented withthe topologi al approa h [9, 10 ℄ (see se tion

2.1), between two dierent ategories : ar and person. We show the possibility of asso iation of

many

R

nodestoone

C

node.Inthe se ondexample,thepresen eof triplet( ar,person,building) is onrmed by a presen e of three ouples of ategories in a lower level that have a spatial ternary

relationship, represented withtheapproa h

∆

-TSR [16 ℄(seese tion2.1), between them.Ingeneral, if the number of ategories is small, the graph is simple. However, the graph an evolve dynami ally,

(14)

by usingdierent relationships :9-area or 16-area relationships for unary relationships on lo ation in

theimage,topologi al relationships or orrelationfor binaryrelationships.

Figure 9 Example of knowledge representation of three ategories at level 2 by using

∆

-TSR relationships thatrepresent ternary relationships between entities.

4 Management of the graph

Inthisse tion,wepresentseveralpossiblemanipulations onourgraph-basedmodel.Theevolution

apa ityof thegraphto thenew knowledgeis representedinse tion 4.1. Ourgraph-based model an

beusedalsotodes ribethe ontext ofanimage.Thisappli ation willbepresentedinse tion4.2. The

advantage ofthis representation is to allow to a elerate thesimilarity omputation oftwo imagesby

(15)

Se tion 4.1.1 dis uss the operation to update the node status. Se tion 4.1.2 present how to infer

thenewknowledge froman existingone.

4.1.1 Update of node status

It is evident that a knew knowledge may hange the status of a node. This modi ation an

have some onsequen es. Here, we say thata node degrades its statuswhile thevalue of its status is

de reased;onthe ontrary,anodeupgradesitsstatus.Forexample,a

C

K

nodedegradesitsstatusfrom true downto false and then upgrades its statusfalse upto un ertain. Here, we denote

DG

and

U G

theoperations of statusdegradation/upgradation ofa node respe tively. There aretwo s enarios ofstatusmodi ation, one for

C

nodesand one for

R

nodes:

2. The se ond ase is when a node

C

K

realises a

U G

. Here, we an have a paradox while a higher level

C

node upgrades its status to a status higher than lower level

C

node's one. To respe t theimposed onstraints, should thelower level

C

Onthe ontrary,we thinkthatstatusmodi ationof

C

nodesinthegraph.

Insummary,onlythestatusmodi ationofa

C

node an impa tthestatusofevery

R

or

C

nodes of lower/higher levels having a link with it. We an develop a re ursive status modi ation of some

(16)

Data:

C

K

,

newStatus

Result: downgrade

C

K

.stat

downto

newStatus

C

K

.stat ←− newStatus

for(

R

J

|(

R

J

.lev = C

K

.lev ∨ R

J

.lev = C

K

.lev + 1) ∧ ∃e(C

K

, R

J

)

) do if (

R

J

.stat > C

K

.stat

) then

R

J

.stat ←− newStatus

if (

∃R

K

|K = J ∧ R

K

.type = R

J

.type ∧ R

K

.stat = R

J

.stat

) then

R

J

←− merge(R

J

, R

K

)

end

for(

C

Z

|(

Z ⊃ I

) do

if (

C

Z

.stat > C

K

.stat

) then

DG(C

Z

, newStatus)

end

Algorithm 1:

DG

degradation algorithm of

C

K

node.

Data:

C

K

,

newStatus

Result: upgrade

C

K

.stat

upto

newStatus

lowestStatus ←−

false.

val

for(

C

J

|(

J ⊂ I

) do

if (

C

J

.stat < lowestStatus

) then

lowestStatus ←− C

J

.stat

end

if (

newStatus 6=

true.

val ∧ newStatus > lowestStatus

) then return;

end

C

K

.stat ←− newStatus

if (

newStatus > lowestStatus

)then //

newStatus =

true.

val

Algorithm 2:

U G

upgradation algorithm of

C

K

node.

nodesfrom a

C

node status hange.

Letusstudythe omplexityofea hoperation.Let

ontrary,andegradationoperationofa

C

nodeatalevel

l

impa tsthe

C

and

R

nodesofhigherlevels. Its omplexity anrea h

O((N

R

₎

(N

L

_−l)

+ (N

C

₎

(N

L

_−l−1)

(17)

Letus examine the example of Figure7 p.10.Supposethat the

C

car

nodedegrades its status(see Fig.10p.16) :

1.

C

car

node degrades its statusto un ertain.

2. Consequently,respe ting onstraintsonthe status ondu ts thateverytrue

R/C

nodes on er-nedby armust hange their statuses to un ertain.

node assigned to ( ar-person) degrades its statusdownto false.

4.1.2 Inferen e knowledge

The rst utilityof our graph is to represent a setof knowledges on entity ategories and on their

relationships. From an image database,after the analysis of its visual ontent, we an onstru t su h

knowledgegraph. Thisgraph an helpus to save,to organize, and to inferall statisti alinformations

on the trends of spatial relationships involving entity ategories ee tively en ountered in the

data-base,withtheaim ofexploitingthem infuture CBIRappli ations, for improvingtaskssu hasobje t

l

i=0

(|K|)!

(|K|−i)!

.

statusmodi ation.Relationships arespatialbinaryrelationships des ribedwitha topologi almodel.

doa re ursive impli ation,thus,the omplexityof thisinferen e is

O(1)

.

4.2 Graph-based representation of image ontext

In se tion 2.1, we have dened the ontext of an image as the representation of every ategories

inthis image and of their spatial relationships. Thus, the graph on ept dened in previous se tions

anbeappliedto des ribethe ontextofanimage. Forexample,the ontext oftheimageinFig.15(a)

an be represented by the graph shown in Fig.16. In this image, four entity ategories : sky, sun,

sea, mountain are dete ted. Suppose that we study only topologi al relationships. Note that the

existen e of the ategory nodes in level

1

allows to extend this graph to a higher level by studying ternaryrelationships. In this se tion, we explain howto ompare the ontext of two images basedon

graphrepresentation.

Se tion 4.2.1denesthesimilaritybetween 2 ategorynodes,se tion4.2.2denestheonebetween

(20)

(b)Inferen esat step1 :inferen esof the

C

nodes

( )Inferen esat step2 :inferen es ofthe

R

nodes

Figure13Exampleofautomati inferen esinthegraph.Thedis ontinuouslinere tanglesrepresent

thesets ofnew nodesinferredfrom existingnodes.

4.2.1 Similarity between two ategory nodes

We an say that two similar ontexts ne essarily ontain ommon entity ategories. Therefore,

rstly, to evaluate the similarity of ontext of images

I

i

and

I

j

, we evaluate the similarity of their entity ategories.

(21)

(b)Inferen e of newun ertainnodes.

Figure 14 Exampleof automati inferen esinthegraph.

(a) (b)

j

)

the set of mat hed ategory ouples that arein ommon between

I

i

and

I

j

:

SC(G

i

, G

j

) = {(C

G

i

K

, C

G

j

K

)}

.We splitthis set intwo subset :aset of (True, false) onrmed ouples (

SC

T F

)and a setof un ertain ouples (

SC

U

). Our

ideaisthatthe omputationof thesimilaritybetween two graphsisbasedon theweightedevaluation

ofthese two sets.Let:

SC

U

_(G

i

, G

j

) = {(C

_K

G

i

, C

G

j

K

)|C

G

i

K

.stat =

un ertain

∨C

G

j

K

.stat =

un ertain

}

SC

T F

_(G

i

, G

j

) = {(C

_K

G

i

, C

G

j

K

)|C

G

i

K

.stat 6=

un ertain

∧C

G

j

(22)

(23)

Thus,

SC(G

i

, G

j

) = SC

U

_(G

i

, G

j

) ∪ SC

T F

(G

i

, G

j

)

. Let

u = |SC

U

_(G

i

, G

j

)|

,

tf = |SC

T F

_(G

i

, G

j

) =

2 × (u + tf )

(|G

i

.{C

K

}| + |G

j

.{C

J

}|)

× e

CN

(15) Knowing that:

e

CN

_{= p}

U

_×

P

u

k=1

sim

CN

(SC

k

U

(G

i

,G

j

))

u

+ (1 − p

U

_{) ×}

P

tf

k=1

sim

CN

(SC

k

T F

(G

i

,G

j

))

tf

where :

SC

X

k

(G

i

, G

j

)|X ∈ {U, T F }

isthe

k

th

oupleof entity ategories of

SC

X

_(G

i

, G

j

)

,

G

i

.{C

K

}

the setof the

C

nodesin

G

i

,

p

U

isweight parameteron evaluationof un ertain part.

sim

CN

_(SC

X

k

(G

i

, G

j

)

variesin

[0..1]

,itevaluatesthe similarityof ouples

SC

k

,then

sim

SC

_(G

i

, G

j

)

also variesin

[0..1]

.

sim

SC

_(G

i

, G

j

) = 1

whentwo images ontains the sameset ofentity ategories. If

P

|SC(G

i

,G

j

)|

k=1

sim

CN

(SC

k

(G

i

, G

j

)) = |G

j

.{C

J

}|

, then

I

i

ontains all of ategories of

I

j

. Here, we an dene thesimilarityof two

C

nodes intwo graph as:

sim

CN

(C

G

i

K

, C

G

j

J

) =

(

0

if

K 6= J

,otherwise

sim(C

K

.stat, C

J

.stat)

(16)

sim(C

K

.stat, C

J

.stat)

evaluatesthesimilarity ofstatusof

C

K

and

.stat = true ∧ C

J

.stat = f alse

.

However, ifwe take into a ount otherattributes for ategory node inea h graph, inthis ase, we

have to onsider the omparisonof image visual ontents also.We an redene the

sim

CN

as:

sim

CN

(C

G

i

K

, C

G

j

J

) =

(

0

if

K 6= J

,otherwise

P

|SetOf Attributes|

k

sim(C

K

.attribute

k

,C

J

.attribute

k

)

|SetOf Attributes|

(17)

where

C

K

.attribute

k

isthe

k

th

attribute of

C

K

and

sim(C

K

.attribute

k

, C

J

.attribute

k

)

the simila-rity of the

k

th

attribute between

C

K

and

C

J

. These attributes are status, furthermore, they an be thesize, shape,or olor histogram of

I

(inthis ase, the similarity is omputed fromthe interse tion oftwo histograms oftwo nodesor anyusual des riptionofgraph).

Themat hingof ategorynodesintwographsisnot omplex,it anberealizedbasedontheir

id

and byusingahashtable.Thisoperationhasa omplexityof

O(N )

with

N = min(|G

i

.{C

K

}|, |G

j

.{C

J

}|)

.

(24)

The omparison of ontext of two images an be studied deeper by taking into a ount

R

nodes in thetwo graphs of these images. Then, similarly to the omparison of

C

nodes inthe two graphs, we dene the similarity fun tion to ompare

R

nodes. We denote

SR(G

i

, G

j

)

the set of mat hed relationship node ouples from

I

i

and

I

j

:

SR(G

i

, G

j

) = {(R

G

i

K

, R

G

j

K

)|R

K

G

i

thenumber of

R

nodes in

G

i

and

N R

G

j

thenumber of

R

nodes in

G

j

that parti ipate to the mat hing. Consequently,we all

sim

SR

_(G

i

, G

j

)

therelationship similarityfun tionoftwo graphsthat an be dened as:

sim

SR

(G

i

, G

j

) =

N R

G

i

_{+ N R}

G

j

|G

i

.{R

K

}| + |G

j

.{R

J

}|

× e

RN

(18) Knowing that:

e

RN

= p

U

×

P

u

k=1

sim

RN

(SR

U

k

(G

i

,G

j

))

u

+ (1 − p

U

) ×

P

tf

k=1

sim

RN

_(SR

T F

k

(G

i

,G

j

))

tf

sim

RN

_(SR

k

(G

i

, G

j

)

is the fun tion that allows to evaluate thesimilarity of two

R

nodes. Then, we an take into a ount any information fromthese nodes su hasfrequen y of ea h

conf ∈ Φ

.

4.2.3 Similarity between two image graphs

Finally,we anevaluatethesimilarityoftwoimagegraphbasedonthesimilarityof ategorynodes

and relationship nodes. We all

SIM (G

i

, G

j

)

thefun tion evaluatingsimilarity between two graphs. Then:

SIM (G

i

, G

j

) = p × sim

SC

(G

i

, G

j

) + (1 − p) × sim

SR

(G

i

, G

j

)

(19) Knowing that

p

isa weight parameter and

SIM

variesin

[0..1]

.

For example,we ompare the ontextsofthetwo imagesfromFig.15. Supposethatwe studyonly

the topologi al spatial relationships and hoose

j

) = 1

, we an saythese two imageshave thesame ontext.

We an add other attributes for ategory nodes in ea h graph, for example information on size

to ea h ategory node (sky, sun, sea, mountain). In the rst image, sky and sea o upy around

39.2%

and

58.3%

of image respe tively. In the se ond image, they o upy around

45.6%

and

54.2%

respe tively. Supposethat we dene

sim

A

_(C

I

.size, C

J

.size)

as:

sim(C

I

.size, C

J

.size) = 1 −

2 × |C

I

.size − C

J

.size|

C

I

.size + C

J

.size

(25)

C

ategorynode

R

relationshipnode

lev

level ofnode

stat

statusofnode

type

type of relationship

Φ

setof ongurations

F

ID

C

(K)

fun tion omputingthe

id

of the

C

node froma setofentity ategories

F

ID

R

(K)

fun tion omputingthe

id

ofthe

R

nodefrom a setofentity ategories

U G/DG

upgrade/downgrade the statusofa node

SC(G

i

, G

j

)

setofmat hed ategory ouples fromgraphs

G

i

and

G

j

SC(G

i

, G

j

)

setofmat hed relationship ouples from graphs

G

i

and

G

j

sim

CN

_(G

i

, G

j

)

similaritybetween two

C

nodes fromgraphs

G

i

and

G

j

(se tion 4.2.1)

sim

SR

_(G

i

, G

j

)

similaritybetween two

Consequently,weobtain

sim

SC

_(G

i

, G

j

) ≃ 0.93

.We anndtooadierents oreifweaddotherspatial relationships like

9DSpa

whi h des ribe dire tionalrelationships [17℄.

5 Spatial reasoning

Inthisse tion,wepresenttwowaysforautomati allybuildinganimage ontextgraphbyusingthe

knowledgegraph:graphbuildingbasedonlyontheknowledgegraphinse tion5.1andgraphbuilding

basedon theknowledge graphand theannotation of image inse tion5.2. An overviewof thesemain

algorithmsproposedispresentedinse tion5.3. Fromthisstrategy,severals enariosofquerybasedon

theimage ontext graph an be applied,we present theminse tion 5.4.

5.1 Image ontext graph building based only on knowledge graph

Wepresenthowtobuildthe ontextgraphofanimage

I

,denoted

G

I

,basedonlyonvisual ontent of

I

andonaknowledgegraph

G

K

.Wesupposethatwehavebuilt

G

K

fromatrainingimagedatabase andthat

G

K

ontains several ategoriesofentitiesandusefulknowledgeonunary,binaryrelationships onthem aswell asontheir visualfeatures (e.g. olor histograms, interest points, et .)or a dedi ated

(26)

from the set of visual attributes of

N

instan es of this one). Note that, every nodes in

G

K

have a true or false statusbe ause we supposethatthetraining database

G

K

representstheuniverseand thatallknowledge olle ted fromthisdatabase isveried.Thepresen eofanentity ategory,a ouple

or a triplet of ategories together at least in an image an be represented by a True

C

node. The relationships between them an be represented by a True

R

node. However, for multiple ategories andagiven ongurationofarelationship type,ifwe annotnd atleastoneexample inthetraining

database,this ongurationis presented inafalse

R

node.

The building of

G

I

is based on the knowledge of

G

K

by exploiting the visual features of

I

, for the onrmation ofthe existen eof anentity ategory in

I

.Dierently to

G

K

,anal

G

.However, this last representation is useless and ompli ate the representationof

G

I

be auseofagreatnumberofabsent ategoriesandthenwedonotputsu hnodes in

G

I

.Consequently,the absen e of

C

node in

G

I

expressestheabsen e ofentity ategoryasso iated withitin

I

.

Algo.3des ribes the main steps ofthis onstru tion. Before startingAlgo.3, we have to dene the

ve following parameters :

1. Type ofunary relationship

type

u

to bestudied in

I

,for example,

type

u

_∈

{9-area,16-area} (see

se tion2.1);

2. Type of binaryrelationship

type

b

to be studied in

I

, for example,

type

b

_∈

{9Dspa,topologi al}

(seese tion 2.1);

3. Probabilitythreshold

ϕ

p

/0 < ϕ

p

≤ 1

:this parameterallowsto lterall ongurations ofagiven type ofrelationship with alowfrequen y or a low presen eprobability in

G

K

.For example,by using unary relationship, in

I

, we an begin by verifying the presen e of a ategory on areas where itspresen eprobability learnt from

G

K

is higherthan

ϕ

p

;

4. Probability threshold

ϕ

t

/0 < ϕ

t

≤ 1

for de iding ofthetrue status ofa ategoryor a relation-shipnodein

G

I

:whenthedete tionprobabilityofanentityofgiven ategory

cat

i

rea hes

ϕ

t

by using thevisual features of an area in

u

/0 < ϕ

u

G

I

is ompletely studiedbyusing

G

G

I

.

ComplexityofAlgo.3, Algo.4,Algo.5:Thethreealgorithmsarepresentedwiththeparadigm

of Obje t-oriented programming (OOP), they are entirely implementable and appli able. Let study

the omplexities ofthese algorithms:

N

Cat

denotes the numberofentity ategories presentedin

G

K

,

N

Cat

Cat

Avg

)

2

2 × (N

Entity

Avg

)

2 )

. Then, Algo.3 has a maximum omplexity of

O

1 = O(N

Cat

_{× O}

2 + O

3 )

. For example,fromthe stati al studies onthesymboli database studied in[15℄,we have :

N

Cat

_{= 86}

,

N

Cat

Avg

= 5 N

Entity

G

I

representing the ontext of

I

G

I

←− ∅

;#Initializethe ontextgraphof

I

L

CN

←− G

K

.get

_

ListOf

_

T rueCN ode

_

AtLevel(0)

; #Getthesetoftrue ategorynodesatlevel0from

G

K

while

L

CN

6= ∅

do

cn

check

←− getRandom

_

CN ode

_

F rom(L

CN

)

; #

cn

check

isa

C

nodeof

G

k

tobelo atedin

I

rn

type

_check

u

←− cn

check

.getRN ode

_

ByT ypeAndStatus(type

u

_,

true

)

; #Getatrue

R

nodeoftype

type

rn

type

u

f oundT rueCN ←− checkCN (I, G

I

, G

K

, cn

check

, type

u

, listConf, ϕ

p

, ϕ

t

, ϕ

u

)

;

#Verifyif

cn

check

anbepresentin

G

I

.Thestatusof

G

I

.cn

check

dependson

ϕ

t

,

ϕ

u

.

checkCN ode

isdenedin Algo.4

if

f oundT rueCN = true

then

Removefrom

L

CN

other

C

nodesthat annot o urwith

cn

check

;

#If

cn

check

ispresentasaTruenodein

G

I

,weremoveall

C

nodesin

G

K

thatare onne tedto

cn

check

byonlyfalse

R

nodes.

1

end

L

CN

.remove(cn

check

)

;#Sin e

cn

check

is he ked,we andeleteitfrom

L

CN

end

buildBinaryRelation(G

I

, type

b

)

;

#Studyotherspatialrelationshipsfrom

G

I

su hasbinaryrelationships,seethisfun tioninAlgo.5 Return

G

I

;

Algorithm 3:Algorithm of ontext graph

G

I

building for an image

I

from a knowledge graph

G

K

.SeeTab.2on page 39for thedenition ofthemain usedvariables/methods.

1. Westudythenon-o urren einthesame imageoftwolevel-0 ategorynodesin

G

K

.Thedeterminationofthis situationisqui klydonebyexaminingthestatusofthe

C

nodeatlevel

1

thatis onne tedtotheselevel-0ones.Ifthe status ofthis node isFalse,this pair ofnodes annoto ur together.It meansthat every binaryrelationshipnodes

(29)

Input: Image

I

, ontext graph

G

I

,knowledge graph

G

K

, ategorynode

cn

of

G

K

to be lo ated in

I

,type ofunary relationship

type

u

,listof ongurations

listConf

to be veried on

cn

,threshold parameters :

ϕ

p

,

ϕ

t

,

ϕ

u

Output:

f oundT rueCN

indi ates ifwe nda true

C

node

f oundT rueCN ←− f alse

;

#Inunaryrelationship,

conf

indi atestheareaandthepresen eprobabilityatthisareaofentity ategoryrepresented by

cn

for (

conf ∈ listConf ∧ conf.probab >= ϕ

p

)do

res ←− I.detectObject(conf, cn)

; #Thisfun tionreturnsdete tionprobability

res.probab

andpossible lo ationofentity

res.location

in

conf

.Notethat,intheworst ase,thedete tiontooldoesnotprovidethelo ationof thedete tedobje t,andthen

res.location

is

conf

nodeStatus ←−

true; if (

res.probab > ϕ

t

) then

f oundT rueCN ←− true

;#Ifitexistsatleastoneentityof ategory presentedby

cn

in

I

,thestatusof the

C

nodeassignedtothis ategoryin

G

I

mustbeTrue

end

else if (

res.probab > ϕ

u

)then

nodeStatus ←−

un ertain; end

cn

G

I

←− G

I

I

←− G

I

.newCN ode

_

AssignedT o(cn.getCategory(0), nodeStatus)

; #

cn

G

I

isassignedtothesame ategoryas

cn

andhasstatusvalueof

nodeStatus

end

if (

f oundT rueCN = true

) then

cn

rn

G

I

←− G

I

.newRN ode

_

AssignedT o(cn

G

I

.getCategory(0), type

u

_{, nodeStatus)}

;

#

rn

rn

G

I

.addConf (conf )

;

G

I

.addM etaData(cn

G

I

, res.location, nodeStatus)

; #Weaddthemetadata on erninglo ationandstatusof

cn

G

I

to

G

I

,thisinformation anbeusedtostudybinary orternaryrelationshipsbetween ategoriesin