Embodied Agents for Multi-party Dailogue in Immersive Virtual Worlds

Embodied Agents for Multi-party Dialogue in Immersive Virtual Worlds

David Traum

University of Southern California Institute for Creative Technologies

13274 Fiji Way Marina del Rey, CA 90292

[email protected]

Jeff Rickel

University of Southern California Information Sciences Institute

4676 Admiralty Way Marina del Rey, CA 90292

[email protected]

ABSTRACT

Wepresentamo delofdialogue foremb o diedvirtualagents

that can communicate with multiple (human and virtual)

agentsinamulti-mo dal setting, including face-to-facesp o-

ken and nonverbal, as well as radio interaction, spanning

multiple conversations in supp ort of an extended complex

task.Themo delbuildsonpreviousworkinemb o diedagents

andmulti-layerdialoguemo dels, andisb eingdeployed ina

p eacekeepingmissionrehearsal exercisesetting.

1. INTRODUCTION

Immersive virtual worlds oer exciting p otential for rich

interactive exp eriences. Human users can cohabit three-

dimensionalgraphicalenvironmentswithvirtualhumansfor

entertainment, education,andtraining. Theycanhavead-

ventures in fantasy worlds. They canlearn ab out history

orother cultures by exp eriencing life indistant places and

times. They can practice tasks, make mistakes, and gain

exp erience without the consequences of real-world failure.

In all these applications , virtual humans can play a wide

variety of roles, including mentorsand guides, teammates,

companions,adversaries, andthelo cal p opulace.

Perhapsthegreatestchallengeincreatingvirtualhumansfor

interactive exp eriencesis supp orting face-to-facecommuni-

cation among p eople and virtual humans. On one hand,

virtual worlds are an ideal application forcurrent sp oken

languagetechnology: theyprovide amicroworldwherecon-

versation can legitimately b e restricted to the events and

objects within its connes. On the other hand, they raise

issues that have received relatively little attention in com-

putational linguisti cs. First,face-to-face communication in

virtualworldsrequiresattentiontoallthenonverbalsignals

(e.g., gaze, gestures, and facial displays) that accompany

humansp eech.Second,conversationsthataresituatedina

3Dworldraiseahostofissues,includingtheattentional fo-

cusoftheconversants,whetherandtowhatdegreetheycan

seeandhearoneanother,andtherelativelo cationsofcon-

versantsand theobjectsthey arediscussing. Finally, since

therewill typicallyb emultiplerealandvirtualp eople, vir-

tual worlds require supp ort for multi-party conversations,

including theability toreasonab outtheactiveparticipants

in a conversation as well as who else might b e listening.

While there has b een some early work in the area ofem-

b o diedconversationalagents[10,19],andsomeofthiswork

hasaddressedhuman-agentdialogues situatedin3Dvirtual

worlds[31],thereiscurrently nogeneralmo delofsuchdia-

logues.

Inthispap er,wediscussprogresstowardsamo delofmulti-

partydialogueinimmersivevirtualworlds. Ourmo delbuilds

onpriorworkonemb o diedconversationalagents,theso cial

psychology literature onthe nonverbal signals that accom-

pany human sp eech, and mo dels of collab orati ve dialogue

from computational linguistics . Themo del is organized as

asetof dialogue management layers, eachincluding an in-

formation stateandasetofdialogue actsthatchangethat

state. The layers include traditional ones, such as turn-

takingandgrounding,aswellasseveralnovellayersaddress-

ing theissues ofmulti-party dialogue in immersive worlds.

Physicalandverbalactionswilloftencontributetomultiple

layers.

2. EXAMPLE SCENARIO

The test-b ed for our emb o died agents is the Mission Re-

hearsalExerciseprojectatTheUniversityofSouthernCal-

ifornia'sInstituteforCreativeTechnologies. Thesettingfor

this project is a virtual reality theatre, including a visual

sceneprojectedontoan8fo ottallscreenthatwrapsaround

the viewer in a 150 degree arc (12 fo ot radius). Immer-

sive audio software provides multiple tracks of spatialized

sounds,playedthroughtensp eakerslo catedaroundtheuser

andtwosubwo ofers.

Within this setting, a virtual environment has b een con-

structed, representing a small village in Bosnia, complete

withbuilding s,vehicles,andvirtualcharacters. Withinthis

environment has b een placed an Army p eacekeeping sce-

nario: aU.S.Armylieutenantndshimselfinthepassenger

seatofasimulatedArmyvehiclesp eedingtowardsaBosnian

village tohelpaplato on introuble. Suddenly,herounds a

andamedic.

into acivilian vehicle, injuring alo cal b oy (Figure1). The

b oy'smotherandanArmymedicarehunchedoverhim,and

asergeantapproachesthelieutenanttobriefhimonthesit-

uation. Urgentradiocallsfromtheotherplato on,aswellas

o ccasional explosions andweap onsrefromthatdirection,

suggest that the lieutenant send his tro ops to help them.

Emotional pleas from theb oy'smother, as well as a grim

assessmentbythemedicthattheb oyneedsamedevacim-

mediately,suggestthatthelieutenantinsteadusehistro ops

tosecurealandingzoneforthemedevachelicopter.

A preliminary demonstration simulation has b een imple-

mented [36], using Hollywo o d storytelling, combined with

technical exp ertise from USC. Figure 2 shows a small ex-

cerptfromthesimulation script.

Inthis interaction, a numb er of issues arise foremb o died

agents, some going b eyond capabiliti es of current imple-

mentedsystems. First,atabroadlevel,wecanseethatthe

agentsconcernthemselveswithmultiple agentsand multi-

pleconversationsduringthisinteraction. Themainscenein

Figure2concerned theLtandSgt,butthemedic wasalso

brought in, and the mother was an imp ortant overhearer.

Other plato on memb ers and townsp eople may also b e p o-

tential overhearers. There is also a separate conversation

b etweenthePlato onSgtandthesquadleaders,starting at

theend of the excerpt given here. Also, in other partsof

thescenario,theLtengagesinradioconversationswithhis

homebase,anotherplato on,and sometimesamedevache-

licopter. Some oftheseconversations havexed b eginning

andendingp oints(esp eciallytheradioconversations),while

othersaremoreepiso dic,trailingawayasthelo calpurp ose

oftheinteractionisestablished andresolved,andattention

of the parties shifts to other matters. In all cases, agents

mustreasonab outwhotheyaretalkingto,whoislistening,

andwhethertheyareb eingaddressedornot.

Thereisalsoanissue,intheimmersivevirtualworld,ofco-

ordination of sp eechwithother communicative mo dalities.

Inmanycases,gesturesandothernonverbalcuesareimp or-

tantincarryingsomeofthecommunicativefunction. Some

examples herearethewaytheSergeant walksuptotheLt

to initiate conversation, the way thatthe Sergeant glances

atthemedic tosignal thatheshouldtaketheturnandre-

sp ondtotheLt'squestion,andthewaythemedicglancesat

themotherwhileformulating alessdirectanswerab outthe

b oy'shealth|fo cusingontheconsequenceofhiscondition

ratherthandirectlystatingwhatmight b eupsetting tothe

mother.

3. PRIOR WORK

Our work builds on prior work in the areas of emb o died

conversational agents[10]andanimatedp edagogical agents

[19]. Several systems have carefully mo deled theinterplay

b etween sp eech and nonverbal b ehavior [9,11, 8, 28],but

thesesystems havefo cused exclusively on dyadic conversa-

tion, and theydid notallowusers andagents tocohabit a

virtualworld. TheGandalfsystem[11]allowedanagentand

humantocohabitarealphysicalspace,andtousegazeand

gestureto referencean object(i.e.,awall-mounted display

screen) inthat space,butthe agent'spresence was limited

to ahead andhandon a2D computermonitor. Similarl y,

the Reaagent [8]cantransp ort herselfto and into virtual

houses and apartments, and the user can p oint to some

objects within those virtual environments, but the user is

notimmersedintheenvironment,andRea'smovementand

references within those environments is verylimited. The

Cosmo agent [24]includes a sophisticated sp eech and ges-

turegenerationmo dulethatcho osesappropriatedeicticref-

erencesandgesturestoobjectsinitsvirtualworldbasedon

b oth spatial considerations and the dialogue context, but

the agent and itsenvironment are renderedin 2D and the

userdo es notcohabitthevirtualworldwithCosmo.

In contrast, Steve [31, 33, 32] cohabits 3D virtual worlds

with p eople and other Steve agents, so it has addressed

b othmulti-party and immersive asp ects ofdialogue in vir-

SGT WalkuptoLT

LT Sergeant,whathapp enedhere?

SGT Theyjustshotoutfromthesidestreetsir. Gesturingtowardsthecivilia n vehicle

Thedrivercouldn'tsee'emcoming.

LT Howmanyp eoplearehurt?

SGT Theb oy andoneofourdrivers. Gesturingtowardtheb oy

LT Aretheinjuriesserious?

SGT Makeseyecontactwithmedicandno ds

MEDIC Driver'sgotacrackedribbutthe kid's{ Glancing atthemother

Sir,wegottagetaMedevacinhereASAP.

LT We'll getit.

LT Plato on Sergeant,securethearea.

SGT YesSir!

SGT (Shouting) Squadleaders! Raises arm

Listenup! Lo oks aroundatsquadleaders

Figure2: Multi-mo dal,multi-characterinteraction excerpt (many nonverbalb ehaviors omitted).

move around invirtual worlds,they aresensitive towhere

humanusersarelo oking,theycanusegazeanddeicticges-

turestoreferencearbitraryobjectsinthoseworlds,andthey

canusegaze toregulate turn-taking in multi-party (team)

dialogues. However, while Steveincludes adialogue mo del

builtonideasfromcomputationallinguisti cs[33],itfallsfar

shortofthe mo delsin state-of-the-artsp okendialogue sys-

tems. Moreover,themo delfo cusesprimarilyonthecontext

ofdyadic conversations b etween aSteve agent andhis hu-

manstudent;thereisverylittledialoguecontextmaintained

forthemulti-partydialogues b etweenaSteveagentandhis

humanandagentteammates.

Workincomputationallinguisti cshasfo cusedonacomple-

mentarysetofissues: ithaslargelyignoredissuesofemb o di-

mentandimmersioninvirtualworlds,buthaspro ducedrel-

ativelysophisticatedmo delsofsp okendialoguethatinclude

a variety of ho oks formultiple mo dalities. We follow the

frameworkof theTrindi project [23],using dialoguemoves

(in this case, corresp onding to actions) as abstract input

andoutput descriptionsforthedialogue mo delling comp o-

nent. This serves particularly well for considering multi-

mo dalcommunication,sinceitallowsmaximumexibilityof

description,includin gmovesthatcouldb eambiguouslyreal-

izedusingeithersp eechoranothermo dality,movesthatre-

quirerealizationusingacombinationofmultiplemo dalities,

ormovesthatsp ecifyamo dality. Wealsoviewthedialogue

moves (and theaÆliated information states)as segmented

into anumb er of layers, eachconcerning a distinct asp ect

ofinformation state,and usingdierent classesof dialogue

acts. Moreover, thereisno oneto onecorresp ondence b e-

tweendialogueactsandatomiccommunicationrealizations:

a single utterance (or gestural communicative action) will

generally corresp ond to multiple (parts of) dialogue acts,

and it may take several communications (sometimes split

into multiple mo dalities) torealizesomedialogue acts.

Asastartingp oint,weusethedialoguelayersdevelop ed in

theTRAINSandEDISdialoguesystems[39,29,26].These

includedlayersforturn-taking,grounding,coresp eechacts,

andargumentationacts(latertermedforwardandbackward-

lo okingacts[16]). Whilenotfullyimplemented withinnat-

ural language dialogue systems, therehas also b een some

otherworkonotherlayersthatb ecomeimp ortant fordeal-

ing with the multi-character, multi-conversati on domain.

ThisincludesworkbyNovickonmeta-lo cutionary acts,in-

cluding anattention level[27],workby Allwo o dandClark

on basiccommunicative functions [2,13],workbyBunton

interaction management functions [7], and work on multi-

levelgroundinginanextendedmulti-mo daltaskinteraction

[15].

4. CURRENT WORK:

MULTI-MODAL DIALOGUE MODEL

Our dialogue agents are designed to run within the Mis-

sionRehearsal Exerciseenvironment [36,30].Thisenviron-

mentincludesamessage-passingeventsimulator,immersive

sound,andgraphicsincludin gstaticsceneelementsandsp e-

cialeects,renderedbyMultigen/Paradi gm's Vega.

Ouragentmo delisbasedonSteve,asdescrib edin thepre-

vious section. Within the Mission Rehearsal Exercise sce-

nario, Steve (and other agents) are given dynamically an-

imated b o dies fromBoston Dynamics'PeopleShop [6];the

primitive motions werecreatedusing motion capture, and

theSteveagentssequencethesemotions dynamicallyin re-

sp onse to the situation by sending commands to the Peo-

pleShoprun-timesoftware. Themedicandsergeantinclude

expressivefacescreatedby Haptek(www.haptek.com)that

supp ortsynchronizatio noflipmovementstosp eech.Steve's

dialoguemo delandrepresentationoftheinteractional state

isb eing augmentedwiththenewdialoguemo del presented

here.

Our dialogue mo del currently consists of the layers shown

in Figure3. Eachoftheseismo deledfromthep ersp ective

ofanagent involved intheinteraction.

We will rst briey describ e each of these, and then give

detailsofthelayersandhowtheasso ciated actsmayb ere-

alizedusingthepaletteofmulti-mo dalcommunicativeabili-

ties. Thecontactlayerconcernswhetherandhowotherindi-

vidualscanb eaccessibleforcommunication. Mo dalities in-

attention

conversation

{ participants

{ turn

{ initiative

{ grounding

{ topic

{ rhetorical

so cialcommitments(obligation s)

negotiation

Figure3: Multi-party, Multi-conversationDialogue

Layers

attentionlayer concerns the object or pro cess that agents

attendto. Contactisaprerequisite forattention. TheCon-

versationlayermo dels the separatedialogue episo des that

goonthroughouttheinteraction. Aconversationisareied

pro cessentity,consisting ofanumb erofsub-elds. Eachof

theseeldsmayb edierentfordierentconversationshap-

p ening at the same time. The participantsmay b e active

sp eakers,addressees,oroverhearers[14].Theturnindicates

theparticipantwiththerighttocommunicate(usingthepri-

marychannel). Theinitiativeindicatestheparticipant who

is managing the content expressed. The groundingcom-

p onent of a conversation tracks how information is added

to thecommon ground ofthe participants. Theconversa-

tion structure alsoincludes a topicthatgoverns relevance,

andrhetoricalconnectionsb etweenindividu alcontentunits.

Once material is grounded, even as it still relates to the

topic and rhetorical structure of an ongoing conversation,

itisalso addedtothe so cialfabriclinking agents,andnot

partofanyconversation. Thisincludessocialcommitments

|b othobligation s toact orrestrictions on action, aswell

ascommitmentstofactualinformation. Thereisalsoane-

gotiationlayer,mo dellinghowagentscometoagreeonthese

commitments. Wenowturntothelayersinmoredetail.

Thecontactlayerismo delled asavectorforallparticipants

that the agent may interact with, eachelement indicating

whethertheparticipant isincontactin themediasp ecied

ab ove. Therearealsodimensionsforwhethersomeoneisin

contacttosendorreceivecommunications bythismo dality.

Theactionsinuencing thislayeraremake-contact,which

couldb e established by turningon aradio orwalkingover

towithineyecontactorearshot,andbreak-contact,which

couldb eestablished bywalkingoutofhearing,turningout

of view (or moving b ehind something), or turning o the

radio. Contact is not generally realized verbally, although

onemightindicateadesireforcontact,e.g.,byshoutingfor

someonetocomeover. Anexampleofamake-contactaction

isshownattheb eginningofourexampleinFigure2,where

theSergeantwalksovertothelieutenant,toinitiatecontact

ofcontact,thoughalsoincludin g an entry fortheagent it-

self, and attention is a one-way phenomenon, rather than

having (p otentially ) distinct send and receive dimensions.

Theactions aecting this layerare divided into those that

an agent p erformsconcerning its ownattention, andthose

relatedtotheattentionofotheragents. Give-attentionin-

volves paying attentionto somepro cess,p erson, or object,

aswellassignallin gthisattention. Thiscanb eaccomplished

b othverbally (e.g.,saying\yes")ornonverbally (gazing at

the object of attention). Withdraw-attention removes

the current object from the attention entry of the agent.

It canb eimplicit ingiving attention to something else, or

p erformed explicitly, bylo oking away insome cases(other

than whenengagedinconversation andserving someother

purp ose,suchasplanningaturnorindicatingturn-taking).

Request-attention signals to an agent that its attention

is desired { it will also require a give-attention action by

the other agent to change the attentional state. Request-

attentioncanb esignalled byacallontheradio,ashout,or

usinganagent'sname,butalsobygestures,suchasraising

an armorwaving. Arelease-attentionactindicates that

attention isno longer required. It o ccurs by default when

apro cess or action thatis the objectofattention ends. It

canalso b e explicit, in the form of adismissal, or gesture

indicatinglackofattention(lo okingaway).Attentionofthe

released agent maystill p ersist, however, until withdrawn,

orgiventosomething else. Direct-attentionsignals that

attentionshouldb egiventoanotherobjectorevent,rather

than the signaller. This canoften b eaccomplished witha

deicticgesture,orwithanutterancesuchas\lo okup!"

Conversation is often a purp ose forwhich attention is de-

sired. Inthiscase,attentionwillb eassumed(unlessexplic-

itlywithdrawn)forthedurationoftheconversation. There

are also explicit indicators of conversational op enings and

closings [34,22]. Conversations are oftenop ened withver-

bal greetings,butnonverbal actionscanb e veryimp ortant

as well. Kendon found a variety of nonverbal actions in-

volvedintheinitiationofconversation[22].Theinteraction

typicallystartswitha\sighting"b eforetheorientationand

approach. Individual s who do not know each other well

and have no sp ecial reason togreet eachother will \catch

the eye" by gazing longer than normal. Next, a \distance

salutation" involves denite eye contact. This is followed

by an approach, which typically involves gaze avoidance.

Finally, a \close salutation" involves resumed eye contact.

TheBo dyChatsystem[12]wasthersttomo deltheseacts

in animated agents. Conversational op enings and closings

areveryformalizedinthemilitaryradiomo dality,e.g.,say-

ing \out" to close a conversation. Actions suchas op en,

continueandcloseaconversationcanb ep erformedeither

explicitly orimplicitly. Also,thereareactionsformaintain-

ing and changing the presence and status of participants.

Anexamplefromthesampledialogue isthewaythemedic

isbroughtintotheconversationfromb einganoverhearerto

anactiveparticipant.

Turn-taking actionsmo del shiftsin the turnholder. Most

canb erealizedverbally,nonverbally,orthroughacombina-

tion ofthetwo. Take-turnisan attempttotaketheturn

bystartingtosp eak. Request-turn(e.g.,signalled byvar-

ious sp eechpreparation signals suchasop ening the mouth

sp eaker's gaze at phrase b oundaries) is an attempt to re-

questtheturnwithoutforciblytaking it[3].Release-turn

(e.g., signalled by an intonational b oundary tone,removal

ofthe handsfromgesturespace,orasustained gazeatthe

listener at the end ofan utterance) isan attempt to oer

theturnto the listener [3,17]. Hold-turn(e.g.,signalled

verballybyalledpause,ornonverballybygazeaversionat

phraseb oundariesorhandsingesturespace)isanattempt

tokeeptheturnatap oint where thelistenermight other-

wisetake it[3,17]. Thesefour turn-taking actshaveb een

mo deledin emb o died conversational agentssincethe earli-

estsystems [9]. Inmulti-party dialogue, thereisonemore

act: assign-turn (e.g.,signalled verbally byavo cativeex-

pressionornonverballybyasp eaker'sgazeattheendofan

utterance)canb eusedtoexplicitly selectthenextsp eaker

[3]. Among emb o died conversationalagents,only Stevein-

cludesthisact.

WeuseInitiativetomo del howtheagent shouldplancon-

tributions. Even though the turn may shift from sp eaker

to sp eaker, in many parts of a dialogue, a single agent is

controlling the owof the contributions, while others only

resp ond to the initiati ve of the other. For some mixed-

initiativedialogues, initiative may shift from one partici-

pant to another. Initiative is sometimes pre-allo cated by

role forsp ecic tasks. Otherwise, it startswith the agent

who op ened the conversation, and can b e managed with

take-initiative, hold-initiative, and release-initiative

actions. These acts can often b e signalled by p erforming

(only)appropriate core-sp eechactsincontext,e.g.,as pro-

p osedby[41,40].Wearenotcurrentlyconsidering nonver-

balsignalsofinitiative.

Weusethegroundingactsfrom[39,37]:initiate;continue;

repair;request-repair;display;acknowledge;request-

acknowledge,andcancel.Seepreviousworkfordetailsof

allbutdisplay,whichisanexplicitsignalofwhatwasunder-

sto o d(e.g., rep eating aword,p erformingan action), leav-

ingitto theoriginal sp eakerto decideifthis actfunctions

asarepair, request-repair,or acknowledge [20]. Emb o died

conversationalagentstypicallyincludenonverbalactionsfor

request-acknowledge (e.g., gaze at listener at grammatical

pauses) and acknowledge (e.g., gaze at sp eaker and no d),

andsomeincluderequest-repair (e.g.,whensp eechrecogni-

tionfails,Peedytheparrotcupshiswingtohisearandsays

\Huh?" [5]).

Topic actions include start-topicand end-topic. Topic

structurecanalsob ecomplex,whennewtopicsarestarted

b eforeoldoneshavecompleted. Topicshiftsofvarioussorts

can b e signalled explicitly with cue phrases (e.g., \now,"

\anyway"),butalsowithnonverbalcues. Headmovements

cansignaltopicshifts;arelatedsequenceofutterancesbya

sp eakertypically uses thesame basicheadmovement, and

thesp eakerwilloftenemployanewtyp eofheadmovement

tomarkthestart ofanewtopic[21]. Topic shiftsarealso

frequentlyaccompaniedbyshiftsinthesp eaker'sb o dyp os-

ture[21].

Our mo del also includes layers for rhetorical structure of

topicalelementswithinaconversation [25],obligations and

commitments[38,1],andnegotiation[4,35]. Wewill,how-

b ecausetheyhavereceivedafairamountofattentioninthe

previousliterature.

5. PLANS AND EVALUATION

Asidefromanalysis ofthescriptedinteraction, wealsohave

twovenuesforevaluatingthemulti-mo dal,multi-party,multi-

conversation dialoguemo del presented ab ove. First,weare

currentlydevelopingaprototyp eend-to-endsp oken-language

dialogue agent (including alsothe emotionmo dels of[18]),

to b e deployed within the immersive virtual world. The

goal for the prototyp e is to handle at least the fragment

presented in Figure 2, and suitable variations. Given ap-

propriate knowledge, goals, andconnections to thevirtual

b o dy, this agent system should b e able to function as ei-

therthesergeant,medic,ormother(weassume,atleastfor

thepresent,ahumanlieutenant, thesubjectofthetraining

exercise).

We are alsogathering actual p erformance datato comple-

menttheHollywo o d-scriptedinteraction. Wehavedesigned

avariant ofthe mission rehearsal exercisewithacombina-

tionofhumanparticipantsandvirtualcharacters|humans

p ortray not just the lieutenant (who is an external test-

subject),butalsotheSergeant,andotheragentswho com-

municate byradio (commandp ost,other plato on,medevac

pilot). Moreover, thereis a\Wizard" controlling the com-

putational agents(and communicating withthe liveones),

usingaxedpaletteofb ehaviors. Initialpilotshavedemon-

strated the viability of the approach, and wewill so on b e

gatheringusabledataonhowthehumanparticipantsinter-

actwitheachotherandwiththevirtualcharacters. Wewill

use this data to evaluate the dialogue mo dels and inform

futuresystemdevelopment.

Acknowledgements

We would like to thank memb ers of ICT and ISIworking

on the MRE project for interesting discussions ab out the

complexities in this scenario. The work describ ed in this

pap er was supp orted in part by the U.S. Army Research

OÆce under contract #DAAD19-99-C-0046. The content

ofthisarticledo esnotnecessarily reectthep ositionorthe

p olicy of theUS Government, and no oÆcialendorsement

shouldb einferred.

6. REFERENCES

[1] J.Allwo o d.Obligations andoptionsin dialogue.

ThinkQuarterly,3:9{18,1994.

[2] J.Allwo o d,J.Nivre,andE.Ahlsen.Onthesemantics

andpragmaticsoflinguisti cfeedback.Journalof

Semantics,9,1992.

[3] M.ArgyleandM.Co ok.GazeandMutualGaze.

Cambridge UniversityPress,Cambridge,1976.

[4] M.Baker.Amo delfornegotiationin

teaching-learni ng dialogues. JournalofArticial

IntelligenceinEducation,5(2):199{254, 1994.

[5] G.Ball,D. Ling,D.Kurlander,J.Miller,D.Pugh,

T.Skelly,A.Stankosky, D.Thiel,M.vanDantzich,

Agents.AAAI/MITPress,MenloPark,CA,1997.

[6] BostonDynamics.PeopleShop1.4UserManual,2000.

[7] H.Bunt.Interactionmanagement functions and

contextrepresentationrequirements.InProceedingsof

theTwenteWorkshoponLanguageTechnology:

DialogueManagementinNaturalLanguageSystems

(TWLT11),pages187{198,1996.

[8] J.Cassell,T.Bickmore,L.Campb ell,

H.Vilhjalmsson, andH.Yan.Conversationasa

systemframework: Designingemb o diedconversational

agents.InJ.Cassell,J.Sullivan ,S.Prevost,and

E.Churchill,editors, EmbodiedConversational

Agents.MITPress,Cambridge,MA,2000.

[9] J.Cassell,C.Pelachaud, N.Badler,M.Steedman,

B.Achorn,T.Becket, B.Douville,S.Prevost,and

M.Stone.Animated conversation: Rule-based

generationoffacial expression,gestureandsp oken

intonationformultiple conversationalagents.In

ProceedingsofACMSIGGRAPH'94,pages413{420,

Reading,MA,1994.Addison-Wesley.

[10] J.Cassell,J.Sullivan, S.Prevost,andE.Churchill,

editors.EmbodiedConversationalAgents.MITPress,

Cambridge,MA,2000.

[11] J.CassellandK.R.Thorisson.Thep owerofano d

andaglance: Envelop evs.emotional feedback in

animated conversationalagents.AppliedArticial

Intelligence,13:519{538,1999.

[12] J.CassellandH.Vilhjalmsson.Fully emb o died

conversationalavatars: Makingcommunicative

b ehaviors autonomous.AutonomousAgentsand

Multi-AgentSystems,2:45{64,1999.

[13] H.H.Clark.Managing problemsinsp eaking. Speech

Communication,15:243{250,1994.

[14] H.H.Clark.UsingLanguage.Cambridge University

Press,Cambridge, England,1996.

[15] P.Dillenb ou rg,D. Traum,andD.Schneider.

Groundinginmulti-mo daltask-oriented collab oration .

InProceedingsoftheEuropeanConferenceonAIin

Education,1996.

[16] DiscourseResourceInitiative.Standards fordialogue

co dinginnaturallanguage pro cessing.Rep ortno.167,

Dagstuhl-Seminar, 1997.

[17] S.Duncan,Jr.Somesignals andrulesfortaking

sp eaking turnsinconversations.InS.Weitz,editor,

NonverbalCommunication,pages298{311.Oxford

UniversityPress,1974.

[18] J.GratchandS.Marsella. Tearsandfears: Mo deling

emotionsandemotional b ehaviors insyntheticagents.

InProceedingsoftheFifthInternationalConference

Animated p edagogical agents:Face-to-faceinteraction

ininteractive learningenvironments.International

JournalofArticialIntelligenceinEducation,

11:47{78,2000.

[20] Y.Katagiri andA.Shimojima. Display actsin

grounding negotiations.InProceedingsofGotalog

2000,the4thWorkshopontheSemanticsand

PragmaticsofDialogue,pages195{198,2000.

[21] A.Kendon.Some relationshipsb etweenb o dymotion

andsp eech.InA.SiegmanandB.Pop e,editors,

StudiesinDyadicCommunication,pages177{210.

PergamonPress,NewYork,1972.

[22] A.Kendon.Adescription ofsomehuman greetings.In

R.MichaelandJ.Cro ok,editors, Comparative

EcologyandBehaviorofPrimates,pages591{668.

Academic Press,NewYork,1973.

[23] S.Larsson andD.Traum.Informationstateand

dialogue managementintheTRINDIdialoguemove

engineto olkit. NaturalLanguageEngineering,

6:323{340,Septemb er2000.Sp ecial IssueonSp oken

Language DialogueSystemEngineering.

[24] J.C.Lester,J.L.Vo erman,S.G.Towns,andC.B.

Callaway.Deictic b elievabil ity: Co ordinating gesture,

lo comotion, andsp eechinlifelikep edagogical agents.

AppliedArticialIntelligence,13:383{414,1999.

[25] W.C.MannandS.A.Thompson.Rhetorical

structure theory: Atheoryoftextorganization.

Technical Rep ortISI/RS-87-190,USC,Information

SciencesInstitute, June1987.

[26] C.Matheson,M.Po esio,andD.Traum.Mo delling

grounding anddiscourseobligation s usingup date

rules.InProceedingsoftheFirstConferenceofthe

NorthAmericanChapteroftheAssociationfor

ComputationalLinguistics,2000.

[27] D.Novick.ControlofMixed-InitiativeDiscourse

ThroughMeta-LocutionaryActs: AComputational

Model.PhDthesis,University ofOregon,1988.also

available asU.OregonComputer andInformation

Science TechRep ortCIS-TR-88-18.

[28] C.Pelachaud, N.I.Badler,andM.Steedman.

Generating facial expressionsforsp eech.Cognitive

Science,20(1),1996.

[29] M.Po esioandD.R.Traum.Towardsan

axiomatizatio n ofdialogue acts.InProceedingsof

Twendial'98,13thTwenteWorkshoponLanguage

Technology:FormalSemanticsandPragmaticsof

Dialogue,1998.

[30] J.Rickel,J.Gratch,R.Hill,S.Marsella, and

W.Swartout.Stevego estoBosnia: Towardsanew

generation ofvirtualhumansforinteractive

exp eriences.InAAAISpringSymposiumonArticial

IntelligenceandInteractiveEntertainment,March

pro cedural traininginvirtualreality: Perception,

cognition, andmotorcontrol. AppliedArticial

Intelligence,13:343{382,1999.

[32] J.Rickel andW.L.Johnson.Virtualhumansforteam

traininginvirtualreality.InProceedingsoftheNinth

InternationalConferenceonArticialIntelligencein

Education,pages578{585.IOSPress,1999.

[33] J.RickelandW.L.Johnson.Task-oriented

collab oration withemb o diedagentsinvirtualworlds.

InJ.Cassell,J.Sullivan, S.Prevost,andE.Churchill,

editors,EmbodiedConversationalAgents.MITPress,

Cambridge,MA,2000.

[34] E.A.SchegloandH.Sacks.Op eningupclosings.

Semiotica,7:289{327,1973.

[35] C.L.Sidner.Anarticial discourselanguage for

collab orative negotiation.InProceedingsofthe

forteenthNationalConferenceoftheAmerican

AssociationforArticialIntelligence(AAAI-94),

pages814{819,1994.

[36] W.Swartout,R.Hill, J.Gratch,W.Johnson,

C.Kyriakakis,K.Lab ore,R.Lindheim, S.Marsella,

D.Miraglia, B.Mo ore,J.Morie,J.Rickel,

M.Thiebaux, L.Tuch,R.Whitney,andJ.Douglas.

Towardtheholo deck: Integratinggraphics, sound,

characterandstory.InProceedingsof5th

InternationalConferenceonAutonomousAgents,

2001.

[37] D.R.Traum.A ComputationalTheoryofGrounding

inNaturalLanguageConversation.PhDthesis,

DepartmentofComputerScience,University of

Ro chester,1994.Alsoavailable asTR545,

DepartmentofComputerScience,University of

Ro chester.

[38] D.R.TraumandJ.F.Allen. Discourseobligation s in

dialoguepro cessing. InProceedingsofthe32 nd

Annual

Meetingofthe Associationfor Computational

Linguistics,pages1{8,1994.

[39] D.R.TraumandE.A.Hinkelman.Conversation acts

intask-orientedsp okendialogue.Computational

Intelligence,8(3):575{599,1992.Sp ecial Issueon

Non-literal language.

[40] M.A.WalkerandS.Whittaker. Mixedinitiative in

dialogue: Aninvestigation into discourse

segmentation.InProceedingsACL-90,pages70{78,

1990.

[41] S.WhittakerandP.Stenton.Cuesandcontrolin

exp ert-clientdialogues. InProceedingsACL-88,pages

123{130,1988.