A Multilingual Platform for Building Speech-Enabled Language Courses

Proceedings Chapter

Reference

A Multilingual Platform for Building Speech-Enabled Language Courses

RAYNER, Emmanuel, et al.

RAYNER, Emmanuel, et al . A Multilingual Platform for Building Speech-Enabled Language Courses. In: Workshop on Second Language Studies: Acquisition, Learning, Education and Technology (L2WS) . 2010.

Available at:

http://archive-ouverte.unige.ch/unige:39600

Disclaimer: layout of this document may differ from the published version.

1 / 1

MannyRayner 1

,PierretteBouillon 1

,NikosTsourakis 1

,JohannaGerlah 1

ClaudiaBaur 1

,MariaGeorgesul 1

,YukieNakao 2

1

UniversityofGeneva,TIM/ISSCO,

40bvdduPont-d'Arve,CH-1211Geneva4,Switzerland

2

LINA,NantesUniversity,

2,ruedelaHoussiniere,BP9220844322NantesCedex03

fEmmanuel.Rayner,Pierrette.Bouillon,Nikolaos.Tsourakis,Johanna.Gerlahgunige.h,

Maria.Georgesulunige.h, baur5etu.unige.h, yukie.nakaouniv-nantes.fr

Abstrat

WepresentCALL-SLT,agenerimultilingualspeeh-enabled

Open Soure CALLsystem based on the“translation game”

idea of Wangand Seneff, foussing on reentenhanements

whihallowtheinstrutortodeneastruturedlanguageourse

dividedupintoasetoflessons.Eahlessonpiksoutasubset

of the orpususinga ombinationof semantiand syntati

onstraints. Wedesribehowthe “struturedlesson” frame-

workinteratswiththespokenhelpfailitiesofferedbythesys-

tem,andoutlinetheinitialsetsoflessonswehaveonstruted

fortheEnglish,FrenhandJapaneseversionsofCALL-SLT.

IndexTerms:CALL,speehreognition,translation,interlin-

gua,onlinehelp,English,Frenh,Japanese

1. Introdution

Aseveryoneknows,itisalmostimpossibletoahieveanyu-

enyinaforeignlanguagewithoutonversationalpratie;the

most effetivewaytopikupalanguage istospendtimein

aountrywhereitisspoken,or, failingthat,to talkregularly

withanativespeaker. Unfortunately, neitheroftheseoptions

are availableformanylanguage students. Itis onsequently

interesting to explore tehniques forreating automati on-

versationpartners. Thereisaontinuumherewithrespetto

thedegreeoffreevariationpermittedonthestudent'spart. At

oneextreme,whihrepresentsurrentommerialmainstream

systemslikeTellMeMoreandRosettaStone 1

,wehave“losed-

response”systems[1℄.Thesystemindiatestothestudentpre-

iselywhattheyaresupposedtosay;thestudentrepeatsit,and

isthengradedontheirpronuniation.Thoughundoubtedlyuse-

ful,thisislessthanideal.Aspointedoutin[2 ℄,effetivesys-

temsforlanguagelearningneedtoallowthelearnertoprodue

largequantitiesofsentenesontheirown,somethingthatis,by

denition, impossibletorealise inalosedresponseapplia-

tion.

Attheotherextreme,oneantrytobuildafullyintera-

tivesystem,whiharriesoutagenuineonversationwiththe

studentinsomekindofsimulatedenvironment;ahigh-prole

exampleisTLCTS[3℄. Itis, however, extremelyhallenging

tomakeasystemofthiskindadequatelyrobust,andinpartiu-

lartoimplementsufientlypowerfullanguageunderstanding.

Thesystemwedesribehere,CALL-SLT[4℄,explores anin-

termediatesolution.Thishasitsrootsinthe“translationgame”

ideaofWangandSeneff[5℄,whosuessfullyreusedspeeh

1

andlanguagetehnologydevelopedatMITunderotherprojets

[6℄tobuildaspeeh-enabledgameforstudentswhowishedto

pratiseChinese.Intheirgame,thesystemshowsEnglishsen-

tenestothestudent,whohastorespondwithaspokenChinese

translation. Thesystem mathesthe studentresponseagainst

theoriginalprompt,andproduesinformativefeedbak. Most

ofthesubjetswhopartiipatedintheinitialstudywerepositive

aboutthesystem.

CALL-SLTusesasimilarbasistrategy,though manyof

thedetailsaredifferent;inpartiular,weuseanotherapproah

tospeehreognition,presentpromptstothestudentinamore

prinipled way, and inlude ahelpsystem whihallowsstu-

dentstoshare experieneaross languages. InSetion2, we

brieypresentmorebakgroundonthesystem.Therestofthe

paperdesribesnewenhanementsnot desribed inprevious

publiations,whihmakeitpossibletoorganisetheorpusof

examplesusedbythesystemasasetofstruturedlessons,eah

oneorganisedaroundaspeiedsyntatiorsemantitheme.

Wepresentthe “lesson struture”frameworkand outline the

initialsetsoflessonswehavebuiltfortheEnglish,Frenhand

JapaneseversionsofCALL-SLT.

2. TheCALL-SLTSystem

CALL-SLTisanOpenSourespeeh-basedCALLappliation

forbeginningtointermediate-levellanguagestudentswhowish

toimprovetheirspokenueny.Thesystemrunsonamedium-

rangeWindowslaptop;itanalsobedeployedonamobileplat-

form,usingthelient/serverarhiteturedesribedin[7℄,with

performaneidentialtothatofthelaptopversion.Theurrent

versionusesarestaurantdomain,andsupportsEnglish,Frenh,

Japanese,Swedish,ArabiandGermanasL2s,withEnglishor

FrenhastheL1 2

.Voabularyvariesfromaround150toaround

500wordsperlanguage,andoversbasisituationssuhasre-

servingatable,orderingfoodanddrink,askingforthebill,and

soon.Table1showstypialexamplesofoverage.

CALL-SLTleveragesearlierworkonRegulus,aplatform

forbuildingsystems based on grammar-basedspeeh under-

standing [8℄andMedSLT,aninterlingua-based speeh trans-

lationframework[9,10 ℄,todevelopageneriCALLplatform

enteredonthe“spokentranslationgame”idea.Ourinitialex-

perienes,inludinganextensivetestarriedoutonseveralhun-

dredSwisshigh-shoolstudents[4℄,havedemonstratedthatthe

Regulus/MedSLTarhitetureisagoodttothistypeofap-

2

Iwouldlikeaminttea

Ateaandaoffeeplease

Doyouhaveatableforfourpeople

CouldIreserveatableforseventhirty

Doyouaeptreditards

Frenh

Puis-jeavoirunebiere

(CouldIhaveabeer)

J'aimeraisdufromagerap´e

(Iwouldlikesomegratedheese)

Jevoudraisunetabledansleoin

(Iwouldlikeatableintheorner)

Est-equejepourraisvoirlemenu

(CouldIseethemenu)

Japanese

Biirunihaionegaishimasu

(Iwouldliketwobeers)

Madogawanosekiwaarimasuka

(Isthereatablebythewindow)

Betsubetsuniharaemasuka

(Canwepayseparately)

Hahijihankarafutarinoteeburuwoyoyaku

shitainodesuga

(Iwouldliketoreserveatablefortwopeople

forhalfpasteight)

Table1: Examplesof CALL-SLToverage in therestaurant

domain,forEnglish,FrenhandJapanese.

pliation. Inpartiular,thegrammar-basedapproahtoreog-

nition gives a response prole with aurate reognition on

in-grammar utteranes andpoor orno reognition onout-of-

grammarutteranes,automatiallygivingthestudentfeedbak

ontheorretnessoftheirlanguageusage.Also,theplatform's

rapiddevelopmentfailities,basedonsemi-automatispeiali-

sationofgeneralresouregrammars,havemadeiteasytoreate

goodspeehreognisersforourinitialdomain(atouristrestau-

rantsenario),despitetheverylimitedavailabilityoftraining

data.AlthoughthereognisersforthevariousL2languagesare

allbuiltfromdevelopmentorporaofatmostafewhundred

examples,nativespeakerstypiallygetper-sentenesemanti

errorratesofunder10%.

Twootherdifferenesbetween CALL-SLT and the MIT

systemarealsoworthhighlighting.First,oneofthemainweak-

nessesofWang'sandSeneff'sworkisthatpromptsareinthe

student'sownlanguage(theL1). Thishastheundesirableef-

fet oftyingthe languagebeingstudied(theL2) toolosely

totheL1inthestudent'smind,andisquiteontrarytomain-

streamtheoriesoflanguageaquisition. Insteadofprompting

studentwithsentenesintheL1,oursystemshowstheminter-

linguarepresentations;thesearereatedusingsemantigram-

marsbasedonourpreviousworkonhuman-readablerepresen-

tationsofinterlingua[11 ℄.

Seond,insteadoffoussingonasinglelanguagepair,we

thinkoftheproblemmorebroadlyasanativityinthemulti-

linguallanguagelearningommunity.Westruturelearninga-

tivitiessoastoenouragestudentstoontributedatabothinthe

L1andintheL2.Eahstudent'sreordednativespeakerdatais

Figure1:MobileversionoftheCALL-SLTsystem,runningon

aNokiatabletandusingthegraphialinterlingua. Thepito-

rialstringisthegraphialprompt,whihhererepresents“Ask

politelyforsoup”. Thebuttonsonthe rightare,from topto

bottom,“reognise”,“nextprompt”and“help”.

Inmoredetail, the game thatforms the basisof CALL-

SLTisasfollows. Thesystemisloadedwithasetofpossible

prompts,reatedbytranslatingthedevelopmentorpusintothe

interlingua.Eahturnstartswiththestudentaskingforthenext

prompt. Thesystemrespondsbyshowingherasurfaerepre-

sentationoftheunderlyinginterlinguaforthesentenetheyare

supposedtoprodueintheL2.Thisrepresentationaneitherbe

textualorpitorial.Forexample,astudentwhoseL1isFrenh

andwhoseL2isEnglishmightbegiventhetextualprompt

COMMANDER DE_MANIERE_POLIE SOUPE

orthegraphialpromptshowninFigure1. Inbothases,an

appropriateresponse wouldbesomethinglike “CouldIhave

thesoup?”,“Iwouldlikesomesoup”,orsimply“Soup,please”;

thegrammarsupportsmostofthenormalwaystoformulatethis

typeofrequest.

Thestudentdeideswhatsheisgoingto say,pressesthe

“reognise”button,andspeaks. Thesystemperforms speeh

reognitionusingaNuane8.5reognitionpakage 3

ompiled

from a grammar-based language model, translates the result

intotheinterlingua,mathesitagainsttheunderlyinginterlin-

guarepresentationoftheprompt,givesthestudentfeedbakon

the math, andadjusts thelevelof difulty upor down. If

themathwassuessful,thestudent'sreordedspeehisalso

savedforfutureuse.

Thestudentmayaskforhelpatanytime. Thesysteman

givehelpeitherinspeehortextform.Texthelpexamplesare

takenfromthe originalorpus, andanalsobeproduedby

translatingfromtheinterlinguabakintotheL1;speehhelp

examplesarereatedby reording suessfulinterations, or

bydoingTTSontextexamples.

3. Construtinglanguageoursesin

CALL-SLT

AfrequentommentfromearlyusersofCALL-SLThasbeen

thatthesystemwouldbemoreusefulifitinludedstrutured

languageexerises.Overthelastfewmonths,wehaveextended

3

WeareurrentlyportingthesystemsothatitalsorunsunderNu-

ane9.0. Despitethesimilarityofnames,Nuane8.5andNuane9.0

theplatformtoinludeaframeworktosupportthistypeoffun-

tionality.Thebasiideaissimple:weallowtheoursedesigner

todivideupthesetofexamplesintoanumberofpossiblyover-

lappingsubsets,eahonedenedbyalistofsyntatiandse-

mantiriteria. Weallagroupingofthiskindalesson. At

runtime, the studentseesamenuofavailablelessons, and is

abletonavigatebetweenthenusingamenu.Eahlessonisas-

soiatedwithapageofexplanation,whihoversthetopior

topisthatitintrodues.

Initially,wehadplanned todenethe ontent oflessons

in termsofsemantiproperties; forexample, wemight have

alessonthatfoussedontimeexpressions,oronnumbers. A

strategyofthiskindaneasilybeimplementedbydeningon-

straintsontheinterlingualrepresentationsassoiatedwiththe

examples. Thisstrategydidindeedseemappropriateforlan-

guageslikeEnglishandJapanese,whihhaverelativelysimple

syntax.Frenhsyntax,however,isonsiderablymorehalleng-

ing,withoneptslikegrammatialgender,agreementandverb

inetionplayingaruialrole;italsoturnedouttobeuseful

tohavetheoptionofinludinglexialonstraints.

Althoughthis ompliatedthearhiteture ofthesystem,

we deided to allowlesson ontent to be dened using any

Boolean ombination ofsemanti, syntati andlexial on-

straints. Alexialonstraintsisimplementedsimplyasare-

quirementthataspeisequeneofwordsshouldbepresent

in the surfae string; similarly, asemantionstraintisa re-

quirementthatagiven,possiblypartiallyinstantiated,struture

shouldmathsomepartoftheinterlinguarepresentation.

Theleasttrivialquestionwashowtopermitdenitionof

syntationstraints. Aftersomeexperimentation,wedeided

that weouldimplementa sufientlyexpressiveframework

by deningasyntationstraintS tobeapartiallyinstanti-

ated syntatiategoryC

S

. Weparseeah example Eusing

thefeaturegrammarwhihformsthebasisforthereogniser's

languagemodel(f.[8℄),andextratallthesyntatiategories

C(E)fromtheanalysistree.WethensaythatEhassyntati

propertySifthereisatleastoneC(E)whihunieswithCS.

IntheFrenhversion,theframeworkletsusdeneonstraints

like“usesinvertedwordorder”,“inludesanadjetive”or“in-

ludesaverbininnitiveform”;forexample,therstofthese

onstraintsisodedasamathagainstthesyntatiategory

vp:[inv=inverted℄. Similarly, in theJapaneseversion,

weandeneonstraintslike“inludesaounter”or“usesthe

plainformofaverb”.

Atsystembuildtime,eahhelpexampleisanalysedtode-

terminethesetoflessonsitmathes,asfollows.Reordedspo-

kenhelpexamplesarerstreplaedbytheirtransripts,andthe

textformoftheexampleisparsedtoprodueananalysistree

andasemantirepresentation.Syntatiategoriesarethenex-

tratedfromeahparsetree,whilethesemantirepresentation

isonvertedintointerlingualform.Finally,thesetofsyntati

andsemantionstraintsassoiatedwitheahlessonismathed

againsttheextratedinformation,andtheresultisahed.

Itisimportantthatthehelpsystemonlyoffershelpappro-

priatetotheurrentlesson,sinetherewillingeneralbemulti-

plehelpexamplesforanygivenprompt.Forexample,suppose

intheFrenhversionthatthepromptis

ORDER POLITELY SOUP

Inanearlylesson,wherethethemeistolearntoaskforthings

inassimpleawayaspossibleandpratiseusingsingularnouns,

thehelpexamplemightbeareordingofJevoudraislasoupe

onstraints=[request=yes,

fr_singular_noun=yes,

fr_adjetive=no,

fr_voudrais_only=yes,

fr_infinitive=no,

fr_time=no,

food_and_drink=yes,

frenh_number=no,

de=no,

du=no℄,

help_file='sing_nouns_help.txt'℄

).

lesson([lesson_id=times,

onstraints=[request=yes,

fr_time=yes,

fr_inf_libre=yes,

person=no,

frenh_lo=no℄,

help_file='time_help.txt'℄

).

Figure2: TwoexamplesofFrenhlessondenitions. Thefor-

mathasbeenslightlysimpliedforexpositionalreasons.

ismakingrequestsusingquestions,thehelpexamplemightbe

Puis-jeavoirla soupe? (“CouldIhavethesoup?”). Similar

onsiderationsapplyinJapanese.Inanearlylesson,ahelpex-

ampleforsomethinglike

ASK-FOR POLITELY TABLE 2 PERSON

willbeformedusingthebasirequestingonstrution...onegai

shimasu,whileinalateroneitmightusethemoreformal...ga

arimasukaor...-tainodesuga.

Aninterestingpointariseswhenthe userresponseis se-

mantiallyorret,butfailstoobeytheonstraintsassoiated

withtheurrentlesson.ContinuingtheFrenhexampleimme-

diatelyabove,supposethatthelessonis“simplerequestsusing

singularnouns”,andthestudentisgiventhepromptabove. If

sherepliesPuis-jeavoirlasoupe?,herresponseissemantially

orret(itwillproduetheorretinterlingua), but itfailsto

maththeurrentsyntationstraints.Weonsideredtheidea

ofwarningthestudentintheseirumstanes,butthisstrategy

seemsin pratie overlystrit; ourobservationisthat itan

onfusestudents,partiularlywhenitisinappropriatelyapplied

duetoareognitionerror.Intheurrentversionofthesystem,

studentsareallowedtorespondusinganyvalidsyntatiform,

irrespetiveofwhetherornotitonformstothethemeofthe

lesson.

3.1. Initiallessonstrutures

WehaveimplementedinitialsetsoflessonsforEnglish,Frenh

andJapanese. Asalready noted,the verydifferentstrutures

of these three languages mean that the lesson struture dif-

fers signiantly from language to language. English and

Japanesehaverelativelysimplegrammars,andthisisreeted

in an arrangement primarily organised aording to seman-

ti/pragmationsiderations.TheEnglishversionofthesystem

“thankyou”,et).

2. Askingforthings(“Iwouldlike”+anounphrase)

3. Askingforthingsusingquestions(“CouldIhave/doyou

have”+anounphrase)

4. Numbers(“Iwouldliketwobeers/atableforthreepeo-

ple/et”)

5. Times (“Iwouldlike a table forsixo'lok/half past

six/quartertoseven/nineteenhundred/et”)

ThesetofJapaneselessonsissimilartotheEnglishone.Syn-

tax, however, plays alargerrolein Frenh; althoughwestill

havelessonsbasedonthemessimilartothoseintheEnglishand

Japanesesystems,severalmoreareorganizedexpliitlyaround

syntatiissues:

1. Singularnouns. Simplerequestsinvolvingonlysingu-

larnouns, e.g. Jevoudraislelapin(“Iwould-likethe

rabbit”).

2. Pluralnouns. Requests involvingpluralnounsandthe

futuretense,e.g.Jeprendrailesfraises(“Iwill-takethe

strawberries”).

3. Compoundnominals.Nounphraseswithdeora, e.g.Je

prendrailesfruitsdemer(“Iwill-taketheseafood”).

4. Adjetives.Frenhadjetivesagreeinnumberandgen-

der, e.g. Jevoudraisunsteakbienuit(“Iwould-like

a-MASCsteak-MASCwelldone-MASC”)

5. Requestsusingquestions.Moreomplexwaystophrase

requests,e.g. Auriez-vousunebiere(“Mightyouhave

abeer?”) or Quels vinsreommandez-vous? (“What

winesdoyoureommend?”)

6. Innitives.Requestsusingtheinnitiveform,e.g.Puis-

jeavoirunebiere?(“CanIhave-INFabeer?”)

Figure2showsexamplesoftwoFrenhlessondenitions,for

singularnounsandtimesrespetively. Therstisintendedto

bedoneearlyinthe ourse,somostnon-trivialonstrutions

arebloked;theseondisintendedtobedoneneartheend,so

mostonstrutionsareallowed.

4. Summaryandfurtherdiretions

WehavepresentedanoverviewoftheCALL-SLTtranslation

game/onversationpartner,foussingonreentworkwherewe

haveintroduedaframeworkthatpermitstheinstrutortodi-

videupmaterialintoastruturedsetoflessons.Wehaveonly

justbeguntoexperimentwiththisnewfuntionality, anditis

stillverymuhunderdevelopment. Itis,unfortunately,notat

alleasytoevaluateCALLsystemsinafullyobjetiveway[12℄;

thequestionwereallywishtoaddressiswhethertheyhelpstu-

dentslearn,andthisrequiresanelaboratemethodologywherea

groupusingthesystemisontrastedagainstasimilaronethatis

notdoingso.WeareurrentlyworkingonanInternet-enabled

versionofCALL-SLT,whihweexpettohaveoperationalby

Q32010. Thisshouldmakeitmuheasiertoperformevalua-

tionexperiments.

Itislikelythatwewillbythenalsohaveelaboratedtheur-

rentlyverysimplelessonstrutureframework. Ataminimum,

wewillertainlyhaveextendedtheexistingsetoflessons,and

implementedlessonplansforsomeoftheotherCALL-SLTlan-

guages,inpartiularGermanandArabi. Wearealsoonsid-

eringexperimentingwithsomemoreambitiousideas;aparti-

pratiseaspeitopiorsetoftopis,whileexludingothers.

Wewillreportonfurtherprogressattheworkshop,wherewe

willalsobeabletodemothesystem.

5. Referenes

[1℄ F.EhsaniandE.Knodt,“Speehtehnologyinomputer-aided

language learning: Strengths and limitations of a new all

paradigm.”LanguageLearningandTehnology,vol.2(1),pp.45–

60,1998.

[2℄ H.Chen,“EvaluatingvespeehreognitionprogramsforESL

learners,”inPapersfromtheITMELT2001Conferene,2001.

[3℄ W.Johnson,“Serioususeofaseriousgameforlanguagelearn-

ing,” ArtiialIntelligene inEduation: BuildingTehnology

RihLearningContextsthatWork,p.67,2007.

[4℄ M.Rayner,P.Bouillon,N.Tsourakis,J.Gerlah,M.Georgesul,

Y.Nakao,andC.Baur,“Amultilingualallgamebasedonspeeh

translation,”inProeedingsofLREC2010,Valetta,Malta,2010,

http://www.isso.unige.h/pub/lre 2 010 allslt.pdf.

[5℄ C. Wang and S. Seneff, “Automati assessment of student

translations for foreign language tutoring,” in Proeedings of

NAACL/HLT2007,Rohester,NY,2007.

[6℄ D.Goddeau,E.Brill,J.Glass,C.Pao,M.Phillips,J.Polifroni,

S.Seneff, and V.Zue, “Galaxy: Ahuman-language interfae

toon-linetravel information,”inProeedings3rdInternational

ConfereneonSpokenLanguageProessing(ICSLP),Yokohama,

Japan,1994.

[7℄ N.Tsourakis,M.Georgesul,P.Bouillon,andM.Rayner,“Build-

ingmobilespokendialogueappliationsusingRegulus,”inPro-

eedingsofLREC2008,Marrakesh,Moroo,2008.

[8℄ M.Rayner,B.Hokey,andP.Bouillon,PuttingLinguistisinto

SpeehReognition:TheRegulusGrammarCompiler. Chiago:

CSLIPress,2006.

[9℄ P.Bouillon,M.Rayner,N.Chatzihrisas,B.Hokey,M.San-

taholma, M.Starlander,Y.Nakao,K.Kanzaki,andH.Isahara,

“Agenerimulti-lingualopensoureplatformforlimited-domain

medial speeh translation,” in Proeedings of the 10th Con-

ferene of the European Assoiation for Mahine Translation

(EAMT),Budapest,Hungary,2005,pp.50–58.

[10℄ P.Bouillon, G.Flores, M.Georgesul, S.Halimi, B.Hokey,

H.Isahara, K.Kanzaki,Y.Nakao,M.Rayner,M.Santaholma,

M. Starlander, and N.Tsourakis, “Many-to-many multilingual

medial speehtranslation on aPDA,” inProeedings ofThe

EighthConfereneoftheAssoiationforMahineTranslationin

theAmerias,Waikiki,Hawaii,2008.

[11℄ P. Bouillon, S. Halimi, Y. Nakao, K. Kanzaki, H. Isahara,

N.Tsourakis,M.Starlander,B.Hokey,andM.Rayner, “De-

velopingnon-Europeantranslationpairsinamedium-voabulary

medial speeh translation system,” in Proeedings of LREC

2008,Marrakesh,Moroo,2008.

[12℄ J.Nerbonne,“Naturallanguageproessinginomputer-assisted

languagelearning,” inHandbook ofComputationalLinguistis,

R.Mitkov,Ed. OxfordUniversityPress,2003,pp.670–698.