MRML: An Extensible Communication Protocol for Interoperability and Benchmarking of Multimedia Information Retrieval Systems

(1)

Proceedings Chapter

Reference

MRML: An Extensible Communication Protocol for Interoperability and Benchmarking of Multimedia Information Retrieval Systems

MULLER, Wolfgang, et al.

Abstract

While in the area of relational databases interoperability is ensured by common communication protocols (e.g. ODBC/JDBC using SQL), Content Based Image Retrieval Systems (CBIRS) and other multimedia retrieval systems are lacking both a common query language and a common communication protocol. Besides its obvious short term convenience, interoperability of systems is crucial for the exchange and analysis of user data.

In this paper, we present and describe an extensible XML-based query markup language, called MRML (Multimedia Retrieval Markup Language). MRML is primarily designed so as to ensure interoperability between different content based multimedia retrieval systems. Further, MRML allows researchers to preserve their freedom in extending their system as needed.

MRML encapsulates multimedia queries in a way that enables multimedia (MM) query languages, MM content descriptions, MM query engines, and MM user interfaces to grow independently from each other, reaching a maximum of interoperability while ensuring a maximum of freedom for the developer. For benefiting from this, only a few simple design principles have to [...]

MULLER, Wolfgang, et al . MRML: An Extensible Communication Protocol for Interoperability and Benchmarking of Multimedia Information Retrieval Systems. In: SPIE Photonics East - Voice, Video, and Data Communications . 2000.

Available at:

http://archive-ouverte.unige.ch/unige:47860

Disclaimer: layout of this document may differ from the published version.

1 / 1

(2)

Interoperability and Benchmarking of Multimedia Information

Retrieval Systems

Wolfgang Muller

, HenningMuller

, Stephane Marchand-Maillet

, ThierryPun

,

DavidMcG. Squire 1

, Zoran Pecenovic 2

,Christoph Giess 3

,Arjen P.de Vries 4

Computer VisionGroup,Computer Science Department, University of Geneva, Switzerland.

1

Computer Science and Software Engineering, Monash University, Melbourne,Australia

2

LCAV and Ergonomics Group, Ecole Polytechnique Federale de Lausanne, Switzerland

3

Medical and BiologicalInformatics, Deutsches Krebsforschungszentrum, Heidelberg, Germany

4

CWI, Amsterdam, The Netherlands

ABSTRACT

While in the area of relational databases interoperability is ensured by common communication protocols (e.g.

ODBC/JDBCusingSQL),ContentBasedImageRetrievalSystems(CBIRS)andothermultimediaretrievalsystems

arelackingbothacommonquerylanguageandacommoncommunicationprotocol.

Besidesitsobviousshorttermconvenience,interoperabilityofsystemsiscrucialfortheexchangeandanalysisof

userdata. Inthispaper,wepresentanddescribeanextensibleXML-basedquerymarkup language,calledMRML

(Multimedia Retrieval Markup Language). MRML is primarily designed so asto ensure interoperabilitybetween

dierentcontent{basedmultimedia retrievalsystems. Further,MRMLallowsresearcherstopreservetheirfreedom

inextendingtheirsystemas needed.

MRMLencapsulatesmultimedia queriesin awaythat enablesmultimedia(MM) querylanguages,MM content

descriptions,MMqueryengines,andMMuserinterfacestogrowindependentlyfromeachother,reachingamaximum

ofinteroperabilitywhileensuringamaximumoffreedomforthedeveloper. Forbenetingfromthis,onlyafewsimple

designprincipleshavetoberespectedwhenextendingMRMLforone'sprivateneeds. Thedesignofextensionswithin

theMRMLframeworkwillbedescribedindetailinthepaper.

MRMLhasbeenimplementedandtestedfortheCBIRSViper,usingtheuserinterfaceSnakeCharmer. Bothare

partof theGNUprojectandcan bedownloadedatoursite

.

Keywords: CBIRS,MRML,interoperability,communicationprotocol,evaluation

1. INTRODUCTION

Duringthe past decadeinterestand research in CBIR has ourished. Oneof the attractions of CBIRresearch is

thatitismultidisciplinary. Awiderangeofresearchissuesmustbeaddressed: humanperception,human-computer

interaction, interfacedesign, queryrepresentation for content-based search, feature extraction and representation,

indexingstructures,term-weightingschemes,pruningstrategies,performanceevaluationandmore. This richnessis

also oneof thegreat challengesfor CBIRresearch. At present, aresearchernew to the eld mustbuild anentire

systembeforeheorshecanaddresstheaspectoftheproblemwhichinterestshimorhermost. Aresearcherworking

alone or in a small team will havegreat diÆculty constructinga state of the art system during the typical time-

frameof aPhD orpost-doc. MRML, theMultimedia RetrievalMarkupLanguage, providesaframeworkin which

componentsof CBIRsystems can be built and exchanged interoperably. If state of the art components of CBIR

systemsare madeMRML-compliant,all researcherswill benetfrom the opportunity to focus theirinvestigations

ontheirareaofgreatestinterestandexpertise.

Presently, almost everycontent-basedimage retrievalsystem (CBIRS) is ahard-wired connection between an

interfaceand thefunctional partsof aprogram. Some programsprovide easy-to-usewebinterfaces, 1

while others

mailto:Wolfgang.Mueller@cui. unig e.ch

(3)

needtobeinstalledlocally andmaybespecictoparticularoperatingsystems. ThereuseofcomponentsinCBIR,

e.g.ofuserinterfaces,isthus verysparse. Thisisnotonlyatime-consumingproblem,sinceeverythingneedsto be

developedanew foreach system,butitmakesthesharingofuserdataand thecomparison ofsystemperformances

diÆcult.

Inorderto addressthese problems, Y.-C. Chang etal.

3

proposed aquerytaxonomyformultimedia databases.

Theyproposedaninitialformulationoftherequirementsforasystemenablingcommunicationbetweenmultimedia

databasesandclients. However,thisapproachisnotyet translatedinto anextensible protocol.

InthispaperwepresentMRML,anXML-basedmarkuplanguageformultimedia queries. MRMLwasdesigned

to facilitate abottom-up developmentapproach, which separates the communication problem from the search for

the best query language (e.g.

4,5

) for multimedia databases. In other words, not only it is designed to fulll the

short-termneedsoftheimage databaseresearchcommunity,butitisalsodesignedtocaterforitslong-termneeds.

The development of standard query languages, together with standard methods for transmitting queries and

data,canimprovetheinteroperabilityofCBIRSsandthusincreasetheuseandusefulnessofmultimedia databases.

SQL andODBCare examplesof such developmentsforrelationaldatabases. Theaimof MRML,however,is more

similar to that of the DICOM protocol, 6

which promoted the interoperability of medical imaging systems from

dierentvendors. Weaddresstheurgentneedforcommontoolswhichwillfacilitatethedevelopmentandevaluation

of multimedia databasesystems. Bythis means, wefacilitate thedevelopmentof commonbenchmarks forCBIRS

performance,similarthoseusedfortextualinformationretrieval.

7

The query-by-example(QBE) paradigm with relevance feedback (including browsing) is the search paradigm

employedbymostcurrentCBIRSs. WethereforeprovideanextensibleQBEfacilitywithin MRML.Further, some

MRML-complianttoolshavebeendevelopedandmadefreelyavailable

undertheGNUPublicLicense. Theyhave

now been accepted as oÆcial part of the GNU project y

. These are described briey in Section 2, and include a

CBIRsearchengine(Viper),whichactsasaserver,andaninterface(SnakeCharmer),whichactsasaclient. Scripts

(mostlyPerlscripts)havealsobeenmadeavailable,whichmightprovideabasisforthecreationofstandardCBIRS

benchmarks. An overviewof various evaluation methods is given in, 8

where theuse of freely-availableannotated

imagecollections,suchas, 9

astestdatasetsisalsoadvocated.

Inorder tobeusefulforresearch, MRMLneedstobea\livingstandard": researchgroupswillneedto beable

to test and use extensions without having to ask a committee for approval. We therefore employ a development

modelwhichpermitsphasesofindependentgrowthwithsubsequentcodemerging. Insection3,wepresentthemain

features ofMRMLand, in section 4,weshowanexampleof howMRML canbeextended to suit particularneeds

whilestayingcoherentwiththecommonstandard.

Currently we are using MRML as the communicationprotocol for a exible benchmarking system supporting

multiple typesofinteraction,asdescribedinthelastsectionandin anotherarticlewithin thisvolume.

10

2. Viper, CIRCUS AND SnakeCharmer

MRMLwasinitiallydesignedtofacilitatecooperationbetweenresearchgroups. Themainprogramsforourtestbed

originatefromtheEcoleFederalePolytechniquedeLausanne(CIRCUSandSnakeCharmer)andfromtheUniversity

ofGeneva(Viper). Inthistestbed,weuseMRMLtolink asingleinterface(SnakeCharmer)totwodierentCBIRS

(CIRCUSandViper). Alink totheMIRROR DBMS 11

iscurrentlyunderdevelopment.

Viper z

is animage searchengine basedon techniques commonlyused in text retrieval and thus oers eÆcient

access to averylargenumberof possible features (more than 80,000simplecolor and texturefeatures, both local

andglobal). DetaileddescriptionsofViper maybefoundin.

12,13

CIRCUS x

isaserverframeworksupportingmultipleimageretrievalmethodsandalgorithms. Currently4methods

basedonLSI 14

andwaveletdecomposition aresupported.

SnakeCharmeris anMRML-compliantclientapplication. Itiswritten in JAVAforportabilityand oersquery

bymultiplepositiveandnegativeexamples,queryhistory,multiplecollectionandalgorithmselection,ascatterplot

oftheresultsaccordingtovariousaspectsofsimilarityandabasket foruser-selectedimages.

y

www.gnu.org

z

http://viper.unige.ch/

x

(4)

MRML {

isformally specied in.

15

It providesaframeworkthat separates thequery formulationfrom theactual

query shipping. It is designed to markup multi-paradigm queries for multimedia databases. MRML enables the

separationofinterfaceandqueryengineandthuseasestheirindependentdevelopment.

MRMLcanbeembeddedintoanexistingsystemwithlittleeort. First,itisXML-based,meaningthatstandard

parsers canbeused to process thecommunicationmessages. Further, the code for anexample MRML-compliant

CBIRsystemis freely-availableandprovidesthebasicimplementation ofbothendsof anMRML-based communi-

cationtoolkit. MRMLis currentlyin atesting phaseat several universitiesand further applicationsbasedon this

protocol suchasbenchmarksystemsandmeta-queryenginesareunder development.

MRMLisdesignedtoallowextensionbyindependentgroups. Bythismeans,itprovidesaresearchplatformfor

extensionswhichlatermaybecomeapartof commonMRML.

3.1. Design goals of MRML

Itisimportantforthefollowingsectionstokeepinmindtheprioritieswhichwetookintoaccountduringthedesign

ofMRML.

Interoperability: Interoperability is an obvious short term need of the CBIRS community. The fact that the

interface betweenCBIRS client and serverisnotspeciedhampers research. Topicsthat could benet from

interoperabilityinclude:

Meta-query engines query several \normal" query engines and assemble the results.

16

Constructing

a meta-query enginewould require to dene a protocol abstraction layercorresponding to each of the

dierent queryembedded in thesystem. Using acommonprotocol would saveasubstantialamountof

work.

Human-computer-interactionaimsatcomparingtheimpactofdierentuserinterfacesontheperformance

ofidenticalqueryengines,ortestseveralengineswiththeidenticalinterfaces. Inthiscontext,byensuring

thecompatibilitybetweenenginesand interfaces,MRMLwouldeasethistypeofevaluation.

Evaluation ofqueryengines: ThankstoMRML, onecandesignabenchmarkpackagethatconnectstoa

server,sendsasetof queriesandevaluatestheresults.

Extensibility without administrationoverhead: it was ourgoal to provide acommunicationprotocol which

can be extendedwithout having to ask a standardizationbody forpermission. MRML enablesindependent

developmentofextensions. AswewilldescribeinSection4.2weinviteMRMLuserstorendertheirextensions

accessible at http://www.mrml.net/extensions/. Later, stable extensions can be added to new common

versionsofMRML.

Commonlog le format: The whole area of CBIRS is craving for ground truth or other user data. MRML

providesacommon,humanreadable,easytoanalyze,formatforloggingcommunicationbetweenCBIRSclient

andserver. MRMLcontainsamaximumofdatawhichmightbeofinterestforcomputerlearningpurposes. If

needed,extensionsofMRMLcanbedesignedin ordertosendadditionaldata.

Simplicityof implementation: everythingwasdesignedsoastominimizetheimplementationoverheadincurred

when using MRML, while keeping a maximum of exibility. MRML is well{formed DTD-less XML. This

eliminatestheneedforvalidatingparserswhilemaintainingalltheexibilitythatisneeded.

3.2. Features of MRML

MRML-based communications have the structure of a remote procedure call: the client connects to the server,

sends arequest, and staysconnected to the serveruntil the serverbreaks the connection. The servershuts down

theconnection after sendingthe MRMLmessagewhich answerstherequest. This connectionlessprotocol hasthe

advantageofeasingtheimplementationoftheserver.Tolimittheperformancelosscausedbyfrequentlyreconnecting,

it is possible to send several requests as partof a singleMRML message. The extension of MRML to aprotocol

permittingthenegotiationofapermanentconnectionisalsoplanned.

MRML,in itscurrentspecication(andimplementation)state,supportsthefollowingfeatures:

{

(5)

openingandclosing sessionswiththeserver,

selection of a data collectionclassied by queryparadigm; it is possible to request collectionswhich can be

queriedin acertainmanner,

selection and congurationof a queryprocessor, also classiedby queryparadigm; MRML also permitsthe

run timecongurationofmeta-queries,

formulationofQBEqueries,

transmissionofuserinteractiondata.

Thenalfeaturereectsourstrongbeliefthataectivecomputing 17

willsoonplayaroleintheeldofcontent-based

multimediaretrieval. MRMLalreadysupportsthisbyallowingtheloggingofuserinteractiondata.

Graceful degradation: independent development on a commonbase Graceful degradationis thekeyto

successfulindependentextensionofMRML.Thebasicprinciplescanbesummarizedasfollows:

serversandclientswhichdonotrecognizeanXMLelementorattributeencounteredinanMRMLtextshould

completely ignoreitscontents,

extensions should be designed sothat all the standardinformation remains available to the generic MRML

user(seeexamplesin Section4).

Theseprinciples provideguidelinesforindependentextensionsofMRML.

ToavoidconictsbetweendieringextensionsofMRML,andinordertoweplantomaintainorpromoteacentral

databasefortheregistrationand documentationofMRML extensions. This would alsofacilitatethe\translation"

betweenuserlogswhichcontainextendedMRML.

3.3. Logging onto a CBIR sever

LoggingonaCBIRserver,as wellasthecongurationhasbeendescribedin thespecications.

15

3.4. Query Formulation

Thequerystepisdependentonthequeryparadigmsoeredbytheinterfaceandthesearchengine. MRMLcurrently

includesonlyQBE,butithasbeendesignedtobeextensible tootherparadigms.

AbasicQBE queryconsistsof alist of images andthe corresponding relevancelevelsassigned to them bythe

user. Inthefollowingexample,theuserhasmarkedtwoimages,theimage1.jpgpositive(user-relevance="1") and

theimage2.jpgnegative(user-relevance="-1"). AllqueryimagesarereferredtobytheirURLs.

<query-step session-id="1"

resultsize="30"

algorithm-id="algorithm-defa ult" >

<user-relevance-list>

<user-relevance-element image-location="http://viper. unige .ch/ 1.jpg "

user-relevance="1"/>

<user-relevance-element image-location="http://viper. unige .ch/ 2.jpg "

user-relevance="-1"/>

</user-relevance-list>

</query-step>

</mrml>

Theserverwill thenreturntheretrievalresultasalistofimages,againrepresentedbytheirURLs.

Queriescanbegroupedintotransactions.Thisallowstheformulationandloggingofcomplexqueries. Thismay

be applied in systemswhich process asingle query using a varietyalgorithms, such as the split-screen version of

TrackingViper 18

orthesystemdescribedbyLeeet al..

19

It isimportantin these casestopreservein thelogsthe

(6)

Inthissection,wedemonstratetheexibilityofMRML.Afterasmallexamplethatillustratesthedesignguidelines

which should be followed in designing extensions, we show how MRML can be extended towards region queries.

ThenweshowhoweasilyMRMLandMPEG-7multimedia descriptionscanbeinterleaved.

4.1. A region query extension for MRML

Inthepast,mostCBIRsystemsconcentratedonqueriesforcompleteimages. Oneoftherstsystemstouseregions

wasBlobworld.

20

Here,theuserispresentedwithasegmentedimage. He/shecanthenspecifywhichofthesegments

(blobs)isrelevantto thequery.

HerewegiveanexamplehowthiscanbeexpressedinMRML,followedbyanexplanation:

resultsize="30"

algorithm-id="algorithm-default ">

<user-relevance-element image-location="http://viper .unig e.ch /ban knote .jpg "

user-relevance="1">

<cui-relevance-region cui-region-start-x="21"

cui-region-start-y="12"

cui-region-outline="e216s137w21 6n13 7"

cui-user-relevance="1"/>

<cui-relevance-region cui-region-start-x="21"

cui-region-start-y="12"

cui-region-outline="s137e216n13 7w21 6"

user-relevance="-1"/>

</user-relevance-element>

<user-relevance-element image-location="http://viper .unig e.ch /2.j pg"

cui-user-relevance="-1">

<cui-relevance-region region-id="ad8b" user-relevance="-1"/>

</user-relevance-element>

</query-step>

</mrml>

TheexamplecontainsaQBEqueryaugmentedbysegmentinformation. Theuser-relevance-elementscontain

theusualinformationcontainedin auser-relevance-elementplus,cui-relevance-regions. Thebenetofthis,

isthatclientsthatarenotregion{awarestillreceiveandunderstand themain partsquery(gracefuldegradation).

A cui-relevance-region contains either the region-id of a previously specied region, or a chain code 21

givingtheoutline of theregion. Chain codes specify regionsby givingastartingpoint and givingdirections from

thisstartingpoint. Inthecaseoftherstcui-relevance-regionthisis: 137pixelssouth,216east,137northand

216west, markingarectangularregion. Deningtheregion to therightof themoving direction asinside,we can

also describe inverse segmentsas it is done in the second cui-relevance-region. Fig. 1describesthe banknote

contained in the image as relevant, but thebackgroundasnot relevant: requested are banknotes with adierent

background.

4.1.1. Classicalquery languages and MRML

SimplyuseaCDATAsectionandexpressthequeryusingthequerylanguagechosen:

resultsize="30"

(7)

dierentfrom the onepresented(markedblack).

<cui-prolog-query result-container="ResultList" cui-relative-weight="0.1">

<![CDATA[ findall(Image,contains(banknot e,Im age), Resu ltLis t). ]]>

<cui-prolog-query/>

<user-relevance-list cui-relative-weight="0.9">

</query-step>

</mrml>

This exampleperforms a QBEquery using the banknote image shown in Fig.1, but without regions. It also

queriesabaseofprologfactsforimageswhichcontainthePrologatombanknote. Thequeryresultswillbemixed,

weighting the rank in the Prolog query with 0.1, and the rankobtained in the QBE query with 0.9. Of course,

designerswhowanttoshare thisuseofMRMLhavetodocumenttheinternals.

4.1.2. MRMLand MPEG-7

MPEG-7, the multimedia content descriptioninterface, is a large collectionof description schemes, aiming at de-

scribingallaspectsofmultimediadata. Currently,MPEG-7datadescriptionsaregiveninXMLSchema. TheXML

Schemaisconsidered asthereplacementoftheXMLDTD(DocumentTypeDenition),i.e. itspecies agrammar

forthetext. WhiletheDTDisgearedtowardstextmarkup,XMLSchemaisgearedtowardsdatatransferandallows

thedenitionoftypesystems.

WedidnotchooseXML{SchemaasabasisforMRMLasthespecicationofXML{Schemaisnotyetstable,and

|atthetimeofwriting|therearenoparsersthatmeeteveryaspectoftheXML{Schemaspecication. However,

any text which contains XML-Schemais also well{formedXML. As aconsequence, there is no problem in piping

MPEG-7multimedia contentdescriptionsthroughMRML,forexampleinaquery:

resultsize="30"

algorithm-id="algorithm-default ">

<mpeg-7>

</mpeg-7>

</query-step>

</mrml>

Again,aqueryprocessorabletoprocessMPEG-7 willrecognizethempeg-7tagand actaccordingly. Whennot

recognized,the tag is simply ignored. Using this method thestructured annotationDS was demonstrated onthe

(8)

MRML'spreferredmechanismfortransferringbinarydataistosendtheURLwherethedatacanbefound. Binary

data is then retrievedusing the URL.As it is aprimary goal of MRMLto enable thesharing of loggingdata we

suggestto transferbigchunks ofdata asfollows.

Binary data which stays constant overseveral sessions (i.e. images and other media items contained in the

queriedcollection) should be transferedusing theirURL,as describedabove. This keepslog les relativelysmall,

yetdataisaccessibleforeveryone.

Binary datawhichchanges duringthequeryprocess(e.g. alecontaininganexampleimageforaQBEquery

whichisnotaccessiblebytheweb)shouldbetransferredusingtwoattributes. Oneoftheattributesshouldcontain

thebase64-encodedbinarydata,theother onethecorrespondingMIMEtype.

However,inmostcases,itispreferabletodesignproperextensionstoMRMLwhichprovidethebestaccessibility

andreadabilityoftheresultinglogs.

4.2. The MRML development model

Asit hasbeenstatedthroughout thisarticle, MRMLallowseachdevelopertoextendMRMLaccordingtohis/her

needs. Inparticular, these extensionscan coexist,and anotication ofacentral body isnotnecessaryformaking

theseextensionswork. However,tomaximizetheusefulnessofMRMLwearepresentlysettingupadatabasewhich

contains documentation of extensions to MRML. It is also intended to providea forumfor groups which want to

extendMRMLinto similardirections.

We propose to develop extensions to MRML in the following fashion. Search rst the page

http://www.mrml.net/extensions/fordocumentationofextensionswhichmightalreadydowhat youwant.

Ifso,

1. implementtheexisting extension;

2. double-checkwith the author of the existing extension that the documentation hasbeen understood in

therightway;

3. addyournameandyouraÆliationtothelistofpeople/groupswhoareusingthisextensionwhichiskept

onwww.mrml.net.

Ifnot,

1. implementtheextension;

2. submitdocumentationforyourextensionalongwithyournameandyouraÆliationtowww.mrml.net.

Theinformationcontainedonhttp://www.mrml.net/extensions/willbeusefulbothforanalyzinglogsandfor

merging extensions,onceanextensionhasprovenmoreusefulthanothers.

5. FURTHER USE OF MRML

Inthisarticle,wehavepresentedastable,extensibleandusefulframeworkfortheuseinCBIRSandothermultimedia

retrievalsystems. ThefollowingtwosectionsdescribepossibleusesofMRML.

(9)

Only preliminary steps have been taken by the CBIR community towards developing common benchmarks { a

comparison of evaluation techniques may be found in.

22

In Fig. 2 we give an UML sequence diagram which

describesourbenchmarkingsystem.

MRMLservesherefortheseparationbetweenbenchmarkingsystem(BS)and benchmarkedsystem(S).Atthe

beginningof thebenchmark theBS will establishaconnectionto the S usingMRML. Itthen congures asession

withagivenalgorithm (thisisaninputparameteroftheBS).TheRelevanceData Repositorycontainsthe results

ofquerieswhichhavebeenevaluated by handbyanumberoftest users. TheBS nowsendsforeachtest userand

each testquery an MRMLqueryto theS. It then usesthe relevance data for generatinga feedback query, which

againareposed totheS.Usingthese datawecanevaluatethelearningcapacitiesof S.All queryresultsarestored

inaresultrepository,allowingcomparisonofdierentrunsanddierentqueryengines.

ReplacingtheQBEInitalQueryFormulatorandQBEFeedbackGeneratorbyothercomponentspermitsimplement-

ing other feedback strategies. Whilein benchmarking of QBE systems the images are considered either as being

relevant ornonrelevant, in the cases of image browsers this notion is not suÆcient. We therefore to support the

notion of relative relevance. How onecan extendthe benchmarking systemto support this notion of relevance is

describedelsewhereinthisvolume.

10

Result Repository

formulate Feedback Query query-result

query-step

storeResults query-result

query-step

RelevanceData Repository Benchmarked

System

MRML generator/

parser

Dispatcher QBEInitialQueryFormulator

get-algorithms

algorithm-list

configure-session

acknowledge-session-op getConfig

collection-list get-collections

configSession

sendQuery

formulateInitial Query

N times (chosen by the benchmarker)

sendQuery

For each query image, for each test user QBEFeedbackGenerator

isRelevantImage

For each result image

isRelevantImage

Benchmarked system

Benchmarking system MRML connection

Figure2. Sequencediagramforourbenchmarkingarchitecture. Here,MRMLisusedforthecommunicationbetween

benchmarking system and benchmarked system. All components, except for the component marked Benchmarked

Systemare part ofthe benchmarking system.

5.2. Meta query engines

Wearecurrentlyconceivingameta queryenginewhichqueriesMRMLcompliantservers. Weplanto usemethods

similartotheonesdescribedin.

16

(10)

MQE.TheMQE will queryasetofotherqueryengines,QE

1

throughQE

n .

Initialization: when started, the MQE will query QE

1

through QE

n

for the list of available algorithms (query

methods) andcollections (usingtheMRMLsignalsget-algorithms,get-collections).

Opening a session: onconnectionwiththeclient,MQEwillsynthesizealistofcollectionsandalgorithmsavailable

onthedierentQE.Herewehavethechoice betweendierentlevelsofelaboration:

ThemostsimpleMQE willjustgivetheuserthechoicebetweenthedierentQE,handingthroughtheir

choices ofalgorithmsand collections,aswellthepropertysheetsforthecongurationofalgorithms.

A more complexMQE will build acomplex property sheet, enabling the distribution of onequery on

multiple queryprocessorsatonce,givingfullcongurationchoice totheuser.

TheMQE willnotyet openasessionontheQE

1:::n .

Before therst query oftheMQE,theMQE willopensessionsoneachoftheQE

1:::n

that hasbeencongured

bytheuser.

Oneach query the MQE will now pass through the queries (query-step)to each QE congured by the user.

It will receivea query-result asan answerbyeach queryengineandmerge these results,passingthe merged

resultstotheclient.

6. CONCLUSION

ThedevelopmentofMRMLandtherstMRML-complianttoolshaveestablishedacommonframeworkforthefast

developmentofCBIRapplications. Toourknowledge,MRMListherstgeneralcommunicationprotocolforCBIR

actuallyimplemented. Thesourcecodefortheinterfaceandthequeryengineis freelyavailableunder GPL

. This

helps developersofretrievalenginesand developersofuser interfacesto developcomplete systemson thebasis of

existingcomponents. Extensivetestshaveshownthestabilityoftheprotocoland ourtestcomponents.

SinceMRMLisafreeandextensible standard,theavailabilityofmoreapplicationsand toolssupportingsucha

protocolwill furtherfacilitatethedevelopmentofCBIRapplicationssupporting adiversityofqueryparadigms.

Moreimportantin ouropinion isthefact that theadoptionof MRMLwillleadto thepossibilityof comparing

dierentCBIRapplicationsobjectively. WehavedevelopedasystemthatwillmakeiteasytobenchmarkallMRML-

compliantsystemsinawaysimilarthebenchmarkwhichexistinthedatabaseandinformationretrievalcommunities.

Finally,thepossibilityofsharingMRMLuserlogswillprovideausefultoolforthesharingofuserinteractiondata

forlearningandevaluationpurposes.

Acknowledgments

This project is supported by the Swiss National Foundation for Scientic Research under grant number 2000-

052426.97.

REFERENCES

1. \Surmagewebdemo."http://www-rocq.inria.fr/cgi-bin/imedia/surfimage.cgi,1999.

2. \QBIC TM

{IBM'sQueryByImageContent."http://wwwqbic.almaden.ibm.com/~qbic/,1998.

3. Y.-C. Chang, L. Bergmann, J. R. Smith, and C.-S. Li, \Query taxonomy of multimedia databases," in Pan-

chanathanet al..

23

(SPIESymposium onVoice,VideoandDataCommunications).

4. J. Z. Li, M.

Ozsu, D. Szafron, and V. Oria, \MOQL: A Multimedia ObjectQuery Language,"in The Third

InternationalWorkshop onMultimediaInformation Systems,pp.19{28,(Como,Italy),September1997.

5. A. de Vries, \Mirror: Multimedia queryprocessing in extensible databases," in Proceedings of the fourteenth

Twente workshop on language technology (TWLT14): Language Technology in Multimedia Information Re-

trieval,pp.37{48,(Enschede,TheNetherlands),December1998.

6. B.Revet,DICOMCook Book for Implementationsin Madalities,PhilipsMedicalSystems,Eindhoven,Nether-

(11)

TextRetrievalConference,pp.1{23,(Gaithersburg,MD, USA),November1998.

8. H.Muller,W.Muller,D.M.Squire,S.Marchand-Maillet,andT.Pun,\Performanceevaluationincontent-based

imageretrieval: Overviewandproposals,"Pattern Recognition Letters ,2000.

9. \Annotatedgroundtruthdatabase."DepartmentofComputerScienceandEngineering,UniversityofWashing-

ton,http://www.cs.washington.edu/research/imagedatabase/groundtruth/,1999.

10. W. Muller, S. Marchand-Maillet, H. Muller, andT. Pun, \Towardsafair benchmarkfor image browsers," in

InternetMultimedia Management Systems, N.N., ed., vol. 4210of SPIE Proceedings,(Boston, Massachusetts,

USA),November6{82000. (SPIEInformationTechnologies2000).

11. A.deVries,M.vanDoorn,H.Blanken,andP.Apers,\TheMirrorMMDBMSarchitecture,"inProceedings of

25thInternationalConference onVeryLargeDatabases(VLDB'99),pp.758{761,(Edinburgh,Scotland,UK),

September1999. Technicaldemo.

12. H. Muller, D. M. Squire, W. Muller, and T. Pun,\EÆcientaccess methods for content-basedimage retrieval

withinvertedles,"in Panchanathanet al..

23

(SPIESymposiumonVoice,VideoandDataCommunications).

13. D.M.Squire,W.Muller,H.Muller,andJ.Raki,\Content-basedqueryofimagedatabases,inspirationsfromtext

retrieval: invertedles,frequency-basedweightsandrelevancefeedback,"inThe11thScandinavian Conference

onImage Analysis(SCIA'99),pp.143{149,(Kangerlussuaq,Greenland), June7{111999.

14. Z.Pecenovic,\Imageretrievalusinglatentsemanticindexing,"nalyeargraduatethesis,AudioVisualCommu-

nicationsLab,EcolePolytechniqueFederaledeLausanne,Switzerland,June1997.

15. W.Muller,Z.Pecenovic,A.P.deVries,D. M.Squire,H.Muller,andT.Pun,\MRML: Towardsanextensible

standard for multimedia querying and benchmarking {Draft proposal," Tech. Rep. 99.04, Computer Vision

Group,ComputingCentre,UniversityofGeneva,rueGeneralDufour,24,CH-1211Geneve,Switzerland,Octo-

ber1999.

http://www.mrml.net/

16. M. Beigi, A. B. Benitez, and S.-F. Chang, \Metaseek: A content-based meta-search engine for images," in

Symposiumon Electronic Imaging: Multimedia Processing andApplications - Storage and Retrievalfor Image

andVideoDatabasesVI, IST/SPIE'98,San Jose, CA,1998.

17. R.W.Picard,AectiveComputing,MITPress,Cambridge,1997.

18. W.Muller,D. M.Squire,H. Muller, andT.Pun,\Huntingmovingtargets: anextensiontoBayesianmethods

inmultimedia databases,"in Panchanathanet al..

23

(SPIESymposiumonVoice, VideoandDataCommuni-

cations).

19. C.S. Lee,W.-Y.Ma, andH. Zhang,\InformationEmbedding BasedonUser'sRelevance Feedbackfor Image

Retrieval,"inPanchanathanetal..

23

(SPIESymposium onVoice, VideoandDataCommunications).

20. C.Carson,S.Belongie,H.Greenspan,andJ.Malik,\Region-basedimagequerying,"inProceedingsofthe1997

IEEE Conference on Computer Vision and Pattern Recognition (CVPR'97), IEEE Computer Society, (San

Juan,PuertoRico),June1997.

21. H. Freeman, \On the encoding of arbitrary geometric congurations," IRE Trans. on Electronic Computers

EC(10),pp.260{268,1961.

22. H. Muller, W. Muller, D. M. Squire, and T. Pun, \Performance evaluation in content-based image retrieval:

Overviewandproposals,"Tech.Rep.99.05,ComputerVisionGroup,ComputingCentre,UniversityofGeneva,

rueGnralDufour,24,CH-1211Genve,Switzerland,December1999.

23. S.Panchanathan,S.-F. Chang,andC.-C.J.Kuo,eds.,MultimediaStorageandArchivingSystemsIV(VV02),

vol. 3846 of SPIE Proceedings, (Boston, Massachusetts, USA), September 20{221999. (SPIE Symposium on

Voice,VideoandDataCommunications).