Dance Movement Patterns Recognition (Part I)

(1)

Dance Movement Patterns Recognition (Part I)

Israel Solà Cerdán

(2)

EEMCS - HMI 2

Acknowledgments Acknowledgments

• • This project would not has been possible without the This project would not has been possible without the collaboration of our project directors:

collaboration of our project directors:

– – Anton Nijholt Anton Nijholt – – Dennis Reidsma Dennis Reidsma – – Jordi Carrabina Jordi Carrabina

• • Thanks for the received aid of: Thanks for the received aid of:

– – Herwin Herwin van Welbergen van Welbergen – – Ronald Poppe Ronald Poppe

• • Thanks to our family, friends, girlfriends Thanks to our family, friends, girlfriends

^(MLG)^(MLG)

and and partners.

partners.

/ 40

(3)

Contents Contents

• • Introduction Introduction

–– PresentationPresentation –– GoalsGoals

–– Organization of the ExpositionOrganization of the Exposition

• • The Virtual Dancer The Virtual Dancer

• • Parlevision Parlevision

• • Work Environment Work Environment

–– MaterialMaterial –– SetupSetup

• • Previous Study Previous Study

–– Interaction DimensionsInteraction Dimensions –– Interaction TreeInteraction Tree

–– Interaction State MachineInteraction State Machine

–– Variables, Animations & TransitionsVariables, Animations & Transitions –– Possible ImprovementsPossible Improvements

• • The Built System (CV) The Built System (CV)

–– New Added FeaturesNew Added Features

(4)

EEMCS - HMI 4

Contents Contents

• • Introduction Introduction

• • The Virtual Dancer The Virtual Dancer

• • Parlevision Parlevision

• • Work Environment Work Environment

• • Previous Study Previous Study

• • The Built System (CV) The Built System (CV)

–– New Added FeaturesNew Added Features –

– New Created PipelinesNew Created Pipelines –– New Sent VariablesNew Sent Variables

/ 39/ 40

(5)

Presentation Presentation

• • Joint project Joint project

• • The Virtual Dancer The Virtual Dancer

• • Obtaining a greater interaction Obtaining a greater interaction

• • Recognizing dance movement patterns Recognizing dance movement patterns

(6)

EEMCS - HMI 6

Contents Contents

• • Introduction Introduction

• • The Virtual Dancer The Virtual Dancer

• • Parlevision Parlevision

• • Work Environment Work Environment

• • Previous Study Previous Study

• • The Built System (CV) The Built System (CV)

/ 39/ 40

(7)

Goals Goals

1. 1. Starting up The Virtual Dancer application (computer vision part Starting up The Virtual Dancer application (computer vision part and artificial intelligent part)

and artificial intelligent part)

2. 2. Analyzing the application and developing improvements Analyzing the application and developing improvements strategies

strategies

3. 3. Recognition of simple dance movements Recognition of simple dance movements 4. 4. Recognition of dance movement patterns Recognition of dance movement patterns

5. 5. Automatic generation of dance movement search patterns Automatic generation of dance movement search patterns

6. 6. Finding patterns without a reference Finding patterns without a reference

(8)

EEMCS - HMI 8

Contents Contents

• • Introduction Introduction

• • The Virtual Dancer The Virtual Dancer

• • Parlevision Parlevision

• • Work Environment Work Environment

• • Previous Study Previous Study

• • The Built System (CV) The Built System (CV)

–– New Added FeaturesNew Added Features –– New Created PipelinesNew Created Pipelines –

– New Sent VariablesNew Sent Variables

/ 39/ 40

(9)

Organization of the Exposition Organization of the Exposition

• • Virtual Dancer application Virtual Dancer application

• • The computer vision system “ The computer vision system “ ParleVision” ParleVision ”

• • The work environment with the The work environment with the setup application and the used material setup application and the used material

• • The study carried out at the beginning of the project The study carried out at the beginning of the project

• • The implemented modifications in the computer vision system “ The implemented modifications in the computer vision system “ ParleVision” ParleVision ”

• • Dance Movement Patterns Recognition (Part II) Dance Movement Patterns Recognition (Part II) – – HMM HMM

– – Recognizing Simple/Complex Movements Recognizing Simple/Complex Movements

– – Auto Generation of Complex Patterns Graphs Auto Generation of Complex Patterns Graphs

– – Test Bench Test Bench

(10)

EEMCS - HMI 10

Contents Contents

• • Introduction Introduction

• • The Virtual Dancer The Virtual Dancer

• • Parlevision Parlevision

• • Work Environment Work Environment

• • Previous Study Previous Study

• • The Built System (CV) The Built System (CV)

/ 39/ 40

(11)

The Virtual Dancer

(12)

EEMCS - HMI 12

Contents Contents

• • Introduction Introduction

• • The Virtual Dancer The Virtual Dancer

• • Parlevision Parlevision

• • Work Environment Work Environment

• • Previous Study Previous Study

• • The Built System (CV) The Built System (CV)

/ 39/ 40

(13)

Parlevision

(14)

EEMCS - HMI 14

Contents Contents

• • Introduction Introduction

• • The Virtual Dancer The Virtual Dancer

• • Parlevision Parlevision

• • Work Environment Work Environment

• • Previous Study Previous Study

• • The Built System (CV) The Built System (CV)

/ 39/ 40

(15)

Material Material

• • Hardware Hardware

– – 2 computers with the following properties 2 computers with the following properties

•

• Pentium 4 2.8GHzPentium 4 2.8GHz

•

• 512 MB of RAM512 MB of RAM

•• Microsoft Windows XP with the SP 2Microsoft Windows XP with the SP 2

– – 1 camera 1 camera

•• Philips ToUcam XSPhilips ToUcam XS

• • Software Software

– – ParleVision ParleVision – – Virtual Dancer Virtual Dancer

• • Others Others

– – Human Dancer Human Dancer – – A black background A black background

– – t- t -shirts with different colors shirts with different colors

(16)

EEMCS - HMI 16

Contents Contents

• • Introduction Introduction

• • The Virtual Dancer The Virtual Dancer

• • Parlevision Parlevision

• • Work Environment Work Environment

• • Previous Study Previous Study

• • The Built System (CV) The Built System (CV)

/ 39/ 40

(17)

Setup Setup

4 5

LAN

3

6

1 2

Port 7521

1 Computer Vision PC 2 Artificial Intelligent PC 3 Camera

4 User

5 Background

(18)

EEMCS - HMI 18

Contents Contents

• • Introduction Introduction

• • The Virtual Dancer The Virtual Dancer

• • Parlevision Parlevision

• • Work Environment Work Environment

• • Previous Study Previous Study

• • The Built System (CV) The Built System (CV)

/ 39/ 40

(19)

Interaction Dimensions Interaction Dimensions

GOAL OF THE INTERACTION (What we want to do)

ACTIONS

(What to do to get these goals)

CAUSE OF THE ACTION (Why are we doing this action?)

TEMPORAL (When occurs the fact)

(20)

EEMCS - HMI 20

Interaction Dimensions Interaction Dimensions

• • Temporal dimension Temporal dimension

– – Before the song Before the song – – During the song During the song – – After the song After the song – – Between songs Between songs

ACTIONS (What to do to get these

goals)

WAY TO DO THIS ACTION (How we want to do it)

/ 39/ 40

(21)

Interaction Dimensions Interaction Dimensions

• • Goal of the interaction Goal of the interaction

– – To attract the attention To attract the attention – – To entertain To entertain

– – To leave a good memory To leave a good memory

goals)

(22)

EEMCS - HMI 22

Interaction Dimensions Interaction Dimensions

• • Actions Actions

– – To attract the attention To attract the attention

• • To say hello To say hello

• • To invite the user To invite the user

• • To be seen To be seen

– – To entertain To entertain

• • To start the dance To start the dance

• • To encourage the user To encourage the user

• • To do a choreography / chorus To do a choreography / chorus

• • To enrich the dance To enrich the dance

• • To explain some step To explain some step

– – To leave a good memory To leave a good memory

• • To congratulate To congratulate

• • To say goodbye To say goodbye

goals)

/ 39/ 40

(23)

Interaction Dimensions Interaction Dimensions

• • Cause of the action Cause of the action

– – Reactive Reactive

– – Deliberative Deliberative

goals)

(24)

EEMCS - HMI 24

Interaction Dimensions Interaction Dimensions

• • Way to do this action Way to do this action

– – Verbal Verbal

– – Non Verbal Non Verbal

goals)

/ 39/ 40

(25)

Contents Contents

• • Introduction Introduction

• • The Virtual Dancer The Virtual Dancer

• • Parlevision Parlevision

• • Work Environment Work Environment

• • Previous Study Previous Study

• • The Built System (CV) The Built System (CV)

(26)

EEMCS - HMI 26

Interaction Tree Interaction Tree

Interaction

Before the song During the song After the song

To attract the attention

To say hello

Reactive / Deliberative

To invite to dance

To be seen

To entertain

To do a Choreography / Chorus

To enrich the dance

To attract the attention

To be seen

Reactive / Deliberative To entertain

To leave a good memory

To congratulate

To say good bye

Reactive / Deliberative To start the dance

Between songs

To entertain

To encourage the user

To explain some step

Reactive / Deliberative To encourage the

user

To encourage the user

Reactive / Deliberative Verbal / Non

Verbal

Verbal / Non Verbal

Verbal / Non Verbal Verbal / Non

Verbal Verbal / Non

Verbal

Verbal / Non Verbal Verbal / Non

Verbal Verbal / Non

Verbal

Verbal / Non Verbal

/ 39/ 40

(27)

Contents Contents

• • Introduction Introduction

• • The Virtual Dancer The Virtual Dancer

• • Parlevision Parlevision

• • Work Environment Work Environment

• • Previous Study Previous Study

• • The Built System (CV) The Built System (CV)

(28)

EEMCS - HMI 28 Waiting a User

(1)

To be seen (2)

To say hello (3)

To invite to dance (4)

To start the

dance (6) To encourage

the user (5)

Dance time (7) To congratulate

(12)

To say good bye (15) To encourage

the user (13)

To explain some step (14)

CV detection

Pad & CV detection

Yes No

User says Yes User says No

User says No User says Yes

To start the song Chorus detected

Random

Absent-minded user To finish the

song

User go out Random

Interaction State Machine Interaction State Machine

To enrich the dance (9)

To encourage the user (10)

To be seen (11) To do a choreography

(8)

To detect passive movements

/ 39/ 40

(29)

Contents Contents

• • Introduction Introduction

• • The Virtual Dancer The Virtual Dancer

• • Parlevision Parlevision

• • Work Environment Work Environment

• • Previous Study Previous Study

• • The Built System (CV) The Built System (CV)

(30)

EEMCS - HMI 30

Variables, Animations & Transitions Variables, Animations & Transitions

• • What animations we could use in each state What animations we could use in each state

• • New animations New animations

• • Conditions required for changing the state Conditions required for changing the state

• • Available Variables Available Variables

• • New variables New variables

/ 39/ 40

(31)

Contents Contents

• • Introduction Introduction

• • The Virtual Dancer The Virtual Dancer

• • Parlevision Parlevision

• • Work Environment Work Environment

• • Previous Study Previous Study

• • The Built System (CV) The Built System (CV)

(32)

EEMCS - HMI 32

Possible Improvements

• •

AnimationsAnimations

– – To say hello moving the hand, looking at the screen center To say hello moving the hand, looking at the screen center – – To bow to the user To bow to the user

– – To clap to the user To clap to the user

– – To say hello putting the head up To say hello putting the head up

• •

OutputsOutputs

– – That our dancer could speak That our dancer could speak

• •

InputsInputs

– – To recognize users’ To recognize users ’ speech speech – – To recognize head’ To recognize head ’s movements s movements

• •

Pattern RecognitionPattern Recognition