Justification of Answers by Verification of Dependency Relations - The French AVE Task

(1)

dependeny relations - The Frenh AVE task

VéroniqueMorieau,XavierTannier,ArnaudGrappy,BrigitteGrau

LIRGroup-LIMSI(CNRS)

first_name.last_namelimsi.fr

Abstrat

This paper presents LIMSI results in Answer Validation Exerise (AVE) 2008 for

Frenh. Wetestedtwoapproahesduringthisampaign: asyntax-basedstrategyand

amahinelearningstrategy. Resultsofbothapproahesarepresentedanddisussed.

Categories and Subjet Desriptors

H.3[Information Storage and Retrieval℄: H.3.1ContentAnalysis and IndexingLinguisti

proessing ;H.3.3InformationSearhandRetrieval

General Terms

Measurement,Performane,Experimentation.

Keywords

Questionanswering,Syntatianalysis,Mahinelearning.

1 Introdution

ThispaperpresentsLIMSIresultsin AnswerValidationExerise(AVE) 2008forFrenh. Inthis

task,systemshavetoonsidertriplets(question,answer,supportingtext)anddeidewhetherthe

answertothequestionisorretandsupportedornotaordingtothegivensupporting text.

Wetestedtwoapproahesduring thisampaign:

•

A syntax-basedstrategy, where thesystemdeideswhether the supporting textis arefor-

mulationofthequestion.

•

Amahinelearningstrategy,whereseveralfeaturesareombinedinordertovalidateanswers:

preseneofommonwordsinthequestionandin thetext,worddistane,et.

Setions2and3presentrespetivelybothapproaheswhileresultsandommentsonerning

oursystemsandthegeneraltaskaregivenin Setion4.

2 A syntax-based strategy: FIDJI

Mostofquestion-answering(QA)systemsanextrattheanswertoafatoidquestionwhenthis

oneisexpliitlypresentin texts,but intheoppositease,theyarenotableto ombine dierent

pieesofinformationforproduingananswer. FIDJI 1

(FindingInDoumentsJustiationsand

1

(2)

Inferenes), anopen-domain QA systemforFrenh, aims atgoing beyond thisinsuieny and

fousesonintroduingtextunderstandingmehanismsrelyingoninferenes. FIDJIusessyntati

information, espeiallly dependeny relations: The goal is to math the dependeny relations

derived from the question and those of the potential answer, as in [8℄. Figure 1 presents the

arhitetureofFIDJIand itsadjustmentsfor theAVE task. Thesystemis atits beginningand

shouldevolvealotin thefuture.

2.1 Proessing of supporting texts

Oursystem relies onsyntatianalysis provided bySyntex [3℄, adependenyparser forFrenh.

Syntex is used to parse questions as well as the doument olletion from whih answers are

extrated. Syntex outputs are heregiven in an easily readable format. The named entities of

doumentsarealsotagged. FortheAVEtask,supportingtextsareonsideredasdoumentsfrom

whihanswershavetobeextrated.

2.1.1 Syntati analysis

Toapply oursystemto theAVEompetition,allsupportingtextsaresyntatially parsed. The

approahistodetet,foragivenquestion(Q)/answer(A

ave

)/supportingtext(T)tuple,ifallthe harateristis of thequestion Qan be retrievedin the text T. Then, the answerproposed by

oursystem(A

f idji

⁾îsômpared^to Â

ave

^: ^if^A

ave

^=A

f idji

^,^theânswerîs^validatedând ^justied^by

T.TodetermineiftheharateristisofthequestionQanberetrievedintext T,FIDJIdetets

syntatiimpliationsbetweenQandT.There aremainly twoases:

1. ThereisanexatmathingbetweensyntatidependeniesofQandT:theNPwhihunies

withthevariableofthequestionrepresentingtheanswerisextrated:

Example:

Q141: Qui est Lionel Mathis ? (Who isLionelMathis?)

attribut(ANSWER, Mathis)

NNPR(Mathis, Lionel)(proper nounrelation)

Text: Lionel Mathis est un footballeur français né le 4 otobre 1981

à Montreuil-sous-Bois (Frane)(LionelMathisisaFrenhfootballer born...)

attribut(footballeur, français)

attribut(footballeur, né)

attribut(footballeur, Mathis)

(3)

OBJ(Verbe, NP2)

⇒

AUX(être, Verbe) modif_par(Verbe, NP1)

Figure2: Exampleofrewritingrule: ativetopassivevoie.

...

ThelemmawhihunieswiththevariableANSWERofthequestionisfootballeur(football

player) and theextrated NP isfootballeurfrançais (Frenhfootball player). TheNP is

omposedoftheheadanditsbasimodiers(nounomplementsandadjetives).

2. There are syntati impliations between Qand T. Beause of syntativariations, infor-

mationin textsarenotalwaysexpressedin thesamewayasin questions. Thus, reasoning

oversyntatidependenyrelations is essential. Asin [2℄, wehave implemented about 30

rewriting rulestoaountforpassive/ativevoie,nominalization ofverbs[7℄, appositions,

oordinations,et.

Rewriting rulesareappliedtoparsedsupporting texts. Inthisway,whateverthesyntati

form ofthequestion,thesystemislikelytond anequivalentsyntatiformulationin the

givensupporting text.

Example:

Q105: Quelle ville a été seouée par un tremblement de terre le 17 janvier ?

(Whih itywashit byan earthquakeonthe 17th ofJanuary?)

DATE( , 17 janvier)

SUJ(seouer, ANSWER)

AUX(être, seouer)

modif_par(seouer, tremblement)

attribut_de(tremblement, terre)

Text: Le tremblement de terre qui a seoué, lundi 17 janvier à 13 h 31, le

nord de la région de Los Angeles ne serait pas assoié diretement à la

fameuse faille de San-Andreas qui balafre la Californie sur des entaines

de kilomètres.

SUJ(seouer, tremblement) SUJ(seouer, nord)

OBJ(seouer, nord)

⇒

AUX(être, seouer)

modif_par(seouer, tremblement)

DATE( , 17 janvier)

attribut_de(tremblement, terre)

attribut_de(nord, région)

attribut_de(région, los angeles)

Theleftolumngivesthedependenyrelationsofthesupportingtextwhihhavealsobeen

rewritten into passive voie (right olumn). In this example, all relations of the question

math withtherelationsofthesupportingtext.

2.1.2 NamedEntities

Thenamed entities of textsare taggedwith about20named entitytypes(person,organization,

loation, nationality,date, number,et.) [4℄. This tagging,ombinedwiththequestionanalysis,

isusefultohekthemathingbetweenthenamed entitytypeexpetedbythequestionandthe

(4)

didBarbaraHendriksgive herrstonert ofthe year?)

Text: <enamex type="PERSON">Barbara Hendriks</enamex> a donné son premier

onert de l'Année nouvelle à <enamex type="LOCATION-CITY">Sarajevo</enamex>.

Allthesyntatidependenyrelationsofthesupportingtextmathwiththoseofthequestion

andtheexpetedanswertypematheswiththetypeoftheextratedanswerSarajevo.

2.2 Answer extration with FIDJI

Theanswerextrationisbasedonsentene-levelanalysis.Senteneshavingthemaximumnumber

ofdependeniesin ommonwiththequestion(inotherwords: Theminimumnumberofmissing

relations)areonsidered. Foreahsentene:

•

Iftheslotfortheanswerin questiondependeniesisuniedintheandidatesentene,then

theorrespondingwordisextrated(seesetion2.1.1).

•

Ifnot,namedentitieshavingtheexpetedtype(ifexisting)areseletedinthesenteneand

thesentenebefore.

Weightsareattributed in orderto rankanswers;As theyarenot usedin AVE, theyare not

desribedhere.

2.3 Answer validation for AVE: heuristis

Attheurrentstateofoursystem,afewheuristisareusedtovalidateananswer. Thedierent

modules desribedaboveprovideinformationonerning:

•

Insomeases,themathing(or not)between,ontheonehand,theexpetednamed entity

typeandanswertype 2

and,ontheother hand,A

ave

^;

•

The rateof syntatidependeniesfrom the questionthat are alsofound (after rewriting)

in thepassage.

The answertype heking is eient when seeked on a largeolletion of douments. It is

quiteraretobeabletoonrmitinasinglepassage. Forthisreason,wedidnotusethislueat

allinourAVE run 3

. Finally,ananswerwasvalidated onlyif:

1. ItwasalsoananswersuggestedbyFIDJI,

2. TheNEtypewastheproperone,

3. The rate of missing dependenies was under a given threshold. This threshold has been

experimentallyset to 30%bytesting dierentongurationson AVE 2006 and AVE 2007

olletions.

Theseheuristishavebeenhoseninordertomaximizepreision. ForanexerisesuhasAVE,

we think that preision is moreimportant thanreall. The seond run,presentedin Setion 3,

hasbeendesignedinordertoimprovereallrate.

Validated vs. seleted. Ifonlyoneanswerwasapprovedbythesystem,itwasmarkedas

SELECTED. Whenmorethanoneanswerwere validated,thebest one(i.e. theonereturnedat

thebest position byFIDJI)wasmarkedasSELECTEDandtheothersasVALIDATED.

2

Anexpliittypesuggestedbythe question,asprimeministerinWhihprimeminister has.... Again,we

donotenterintodetailsforthispartbeausewedonotuseitinAVE.

3

Butourseondrunpresentedinnextsetionhekstheanswertypeinaverydierentway,throughWikipedia

(5)

thatWikipediaartilesonerningpersonsontainverylong-distanepronominalanaphoras;for

mostofthem, these referenesanberesolvedbyreplaing thepronounbytheartiletitle. We

usedthissimpletrikwithpronounsil (he) andelle(she).

Resultsarepresentedanddisussedinsetion4.

3 FRASQUES as an entry of a mahine learning system

Theseondsystemfollowsamahinelearningapproahandappliesthequestion-answeringsystem

FRASQUES [6℄in orderto omputesomeof thelearningfeatures. Thelearningsetisextrated

fromthedataprovided byAVE2006andontains75%ofthetotaldata.

Thehosenlassierisaombinationofdeisiontreeswiththebaggingmethod. Itisprovided

bytheWEKA 4

programthat allowsto testalotof lassiers.

Thenextsetionspresentthedierentfeatures.

Thespeifeaturesbasedonthevoabularyarepresentedin[1℄,while[5℄showsandevaluates

thesefeaturesandpresentsthemahine learningmethod.

3.1 Common terms

If apassage ontainsmanyterms ofthe questionthen it oughtto beabout thesametopi and

would probablyontain the answer. Thus, the rst feature is the rate of termsof the question

thatareinthesupportingtext,withorwithoutlexialvariations. Thesevariationsarereognized

byFastr[7℄.

Amongquestionterms,someplayamoreimportantroleandaresupposedtobefoundin the

supportingpassagesorhavetobeveried. Fourpartiularrolesaredistinguishedinthequestions:

•

Fous: Thefousistheentityaboutwhihthequestionisaskedandeitheraharateristi

or adenition of this entity hasto besearhed. InWhih is the politial party of Lionel

Jospin?,thefousisLionelJospin.

•

Answer type: Whenthe spei answertype is expliitin a questionand reognized in

a passage, it allows the system to hek that the proposed answer ts the expeted type

by applying somesyntatirules. Intheprevious question,the expeted typeis politial

party.

•

Main verb: Theverbin thequestionthathasanimportantrolebeauseitorrespondsto

anationorafat.

•

Bi-terms: Abi-termismadeoftwowordssyntatiallylinkedasNobelPrize. Ifabi-term

isin thequestionandinthepassage,then thewordsarelikelytohavethesamemeaning.

Eah of these terms onstitutes a feature given to desribe a passage. These elements are

automatiallyreognizedbythequestionanalysismoduleofFRASQUES.

3.2 Answer veriation

AnotherfeatureisbasedontheanswerextratedfromthepassagebyFRASQUES.IftheFRASQUES

answerisequaltotheanswerto judge,thelatterisprobablyorret.

TheextrationstrategyofFRASQUESdependsontheexpetedtypeofanswer. Ifthistypeis

anamedentity,theentityoftheexpetedtypewhihislosestto thequestionwordsisseleted.

Otherwise, patterns of extration are used. These patterns express the possible position of the

answerwithrespet tothequestionharateristissuhasthefousortheexpeted typeof the

answer.

4

(6)

This feature relies on the proximity of the ommon terms. The system looks for the longuest

ommonstringofonseutivewordsin thepassageand thehypothesiswithoutonsidering their

order. Thehypothesisorrespondstothearmativeformof thequestiononatenatedwiththe

answer.

Toomputethishain, thestrategyisthefollowing:

1. In order to failitate the omparison between the text and the hypothesis, the words are

normalized(lemmatizationand bringingof synonymstogether).

2. Thealgorithm looks forthelongestgroupsofadjaentwordsommonto thequestionand

thehypothesis.

3. A string is initialized with eah of these groups. Eah string will grow by onatenating

adjaentgroups(orgroupsthatareseparatebyalloweditemslikeaommaoradeterminant).

Thegroupsanalsobeseparatedbyoneplainwordountingasabonus. Onlyonebonus

isallowed.

For example, when omparing the strings Elisabeth 2, l'atuelle reine (Elisabeth II, the

urrent queen) and Elisabeth 2 reine (Elisabeth II queen), Elisabeth 2 and reine are

joigned beause theyare separatedby a determinant (l' ), a ommaand only oneother

word(atuelle).

4. The longesthain is seletedand thevalueof thefeature is theratiobetweenthenumber

ofwordsin thehainandthenumberofwordsinthehypothesis.

3.4 Cheking the answer type with Wikipedia

Alotofquestionsexpetananswerofaspeiedtype. ForexamplethequestionWhatsportdid

Zinédine Zidane pratie? expets akindofsport asanswer. Toverify thetypeof theanswer,

weusetheenylopaediaWikipedia 5

.

The hypothesis is that if thetype anbefound in the Wikipediapage orresponding to the

answer,weonsiderthattheanswerorrespondsto theexpetedtype.

Themethod looks forthetypein theWikipediapagewhosetitle ontainstheanswer. If the

typeisfound, thevalueof thefeature is1else itis 0. Forquestions withoutexpetedtype, the

valueis-1.

3.5 FIDJI features

Somefeaturesomingfrom FIDJIarealsoadded:

•

Whether FIDJI validated,ignored (known bybelow thethreshold) orrejeted (unknown)

theanswer;

•

Goodorbadnamed entitytype;

•

Rateofmissing dependanies inthepassage.

4 Results and omments

OialresultsforourtworunsaregiveninTables1.a(runbasedonsyntatirelations)and1.b

(ombiningdierentharateristisbytheuseofalassierandinludingassupplementaryfeature

therateofpreseneofdependenyrelationsgivenbyFIDJI).

5

Wikipedia:http://fr.wikipedia.org

(7)

Fmeasure 0.57

PreisionoverYESpairs 0.88

RealloverYESpairs 0.42

qaauray 0.19

estimated_qa_performane 0.29

Fmeasure 0.61

qaauray 0.23

estimated_qa_performane 0.32

Table1: OialAVE 2008results.

Fmeasure 0.63

Table2: Resultswithoutdependenyrelations

Table 2showsthe results obtainedwhen we donot inlude harateristis omingfrom the

FIDJIsystem. Finally, abaselinefor Frenh AVE, provided bythe organizers andpresentedin

Table3,orrespondstothestrategyonsistingin answeringYEStoeahpair.

Now,thequestion isto knowthe signiationof this testset. This year, theFrenhtest set

ismadeof199triples,builtfrom108dierentquestions. Thereare1.8triplesinaverageforeah

question: 39questions have1answerto justify, 47questions have 2answers, and22 questions

have3proposed answers. Theratioofvalidated pairsis 29%,that isto saythat only52 triples

areorret. ThesearetheresultsprovidedbyonepartiipanttothemonolingualFrenhQAtask

andthebilingualtaskswithFrenhastarget. Thus,theanswersresultfromasinglesystem.

If weompare thistest set tothe testset provided forFrenhin 2006, there were 5dierent

systemsthathavegiven3200answersto 190questions: amongthem, 627answerswerejustied.

So,theurrenttestsetouldonlymeasuretheabilityofaAVEsystemtoevaluatetheresults

of one QA system, whih is the best system in this language, but annot allowto measure its

abilityinageneralexerieofanswervalidation. Moreover,aretheresultsreallysigniantwhen

theyarealulatedoveratotalof50? Oneanswerisequivalentto2points.

The goal of an evaluation ampaign is generallytwofold : to provide ressoures allowingto

develop systemsable to solve atask and omparing thedierent approahesdevelopedfor this

task. Inordertotendtowardsthesegoals,theFrenhAVEtestsetouldnotonlybemadeofthe

resultsof theurrent QAtraks. Itoughtto be ompletedso that the numberofexampleswill

besigniantandthephenomenatotreatrepresentativeofatask, andnotofasystem.

Anotherimportantpointonernsthedenitionofwhatajustiationofananswertoaques-

tionis. Whihpieesofinformationmustthepassageontain? InAVE,itseemsthatiftheorret

answerisin thepassage,itisvalidated,evenifthetopi isonlypresentwithan anaphora,asin

thefollowingpair:

Q: Combien laville de Colomboomptait-elle d'habitants en 2001? (How many inhabitants

arethere inColomboin2001 ?)

A:377396

J:Lavilleompte377396habitantsen2001pour2234289dansl'agglomérationet'estlaville

lapluspeupléeduSriLanka,ainsiquele½urdel'ativitéommerialedeepays. (Thetownhas

377396 inhabitantsin2001 ...)

Fmeasure 0.45

RealloverYESpairs 1

(8)

ville(the town) is Colombo,evenifthenameof thedoumentisCOLOMBO.Theonlyname

ofaWikipediapageannotallowtoverify therefereneoftheanaphora.

5 Conlusion

Wehavepresentedin thispapertwostrategiesfordeidingifananswertoaquestionisjustied

byagivenextratoftext.

Therst isbased onasyntatiapproahin orderto verify that notonly thevoabularyis

similar betweenthequestionand thepassage,but also that thisvoabularyis used in thesame

meaning. Thisisdonebyverifyingthesimilarityoftherelationsbetweentheorrespondingterms.

Someheuristisarethenhosenfordeidingifapassagejusties ornotananswer.

Suh an approah has good performanes at the preisionlevel, but the reall remainslow,

beause of errors done by the syntatiparser, as in all these kind of approahes. So, wealso

testedanotherapproahonsistingindeidingifapassageisajustiationornotaordingtoa

setoffeatures. Thedeisionistheresultofalassier,automatiallytrained. Theselastapproah

hasbeendevelopedlastyear,andwehaveaddedthisyearanewfeaturebasedonthedependeny

relations. We have to test our results on other orpora in order to validate the gain that this

featureseemstobringout.

Referenes

[1℄ A.Ligozat, B. Grau, A. Vilnat, I. Robba, and A. Grappy. Lexial validation of answers in

questionanswering.InProeedingsof workshopon WebIntelligene, WI,SilionValley,2007.

[2℄ G. Bouma,I. Fahmi, J.Mur, G. vanNoord, L. vander Plas, and J.Tiedemann. Linguisti

knowledgeand questionanswering. Traitementautomatique deslangues,46,2007.

[3℄ D. Bourigault and C. Fabre. Approhe linguistique pour l'analyse syntaxique de orpus.

Cahiersde Grammaire,25,2000.

[4℄ F. Elkateb. Extrationd'entités nomméespourlareherhed'informationspréise. In4éme

ongrèsISKO-Frane,L'organisation desonnaissanes, Grenoble,2003.

[5℄ A.Grappy,A.Ligozat,andB.Grau.Evaluationdelaréponsed'unsystèmedequestion-réponse

et desajustiation. InProeedings of workshop on COnféreneen Reherhe d'Information

etAppliations,CORIA, Trégastel,2008.

[6℄ B.Grau,A.Ligozat,I.Robba,A. Vilnat,andL.Moneaux. Frasques: A question-answering

systemintheequer evaluationampaign. InProeedings of workshop on Language Ressoure

Evaluation Conferene, LREC,Genoa,2006.

[7℄ C. Jaquemin. A symboli and surgial aquisition of terms through variation. In Con-

netionist,Statistial andSymboliApproahes toLearning for Natural Language Proessing,

Heidelberg,1996.

[8℄ B. Katz and J. Lin. Seletively using relations to improve preisionin question answering.

InProeedings of workshop on Natural Language Proessing for Question Answering,EACL,

Budapest,1999.