Towards a Hand Skeletal Model for Depth Images Applied to Capture Music-like Finger Gestures

HAL Id: hal-00875721

https://hal-mines-paristech.archives-ouvertes.fr/hal-00875721

Submitted on 22 Oct 2013

HAL is a multi-disciplinary open access archive for the deposit and dissemination of sci- entific research documents, whether they are pub- lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Towards a Hand Skeletal Model for Depth Images Applied to Capture Music-like Finger Gestures

Arnaud Dapogny, Raoul de Charette, Sotiris Manitsaris, Fabien Moutarde, Alina Glushkova

To cite this version:

Arnaud Dapogny, Raoul de Charette, Sotiris Manitsaris, Fabien Moutarde, Alina Glushkova. To-

wards a Hand Skeletal Model for Depth Images Applied to Capture Music-like Finger Gestures. 10th

Int. Symposium on Computer Music Multidisciplinary Research (CMMR’2013), Oct 2013, Marseille,

France. �hal-00875721�

123456784894A68BCDEDF4E826DE8258DF84D7 ED68F284F5D87ECD8AD58D7F5D7

1

234567895ABC4DE8

F5B687853E8

B385453E8

1

548B6537E8

24586B5

1

FBB 8!5E89A53"48B#85$"5%688D&"E848'53( E8354

6"758( 4BBCD85478B"A63835A 8!5E89A53"48B#82AA78)4#B3"5 E8

*43D8B#85 7B45E83

+534567,75ABC4DE835B6-7- 53E8B3,"5453E8#54,"B6537./"40A53 ,#3 545,C6B5/6B",76,C3

7F54F!8 (8)454C866358135C82)138"A8C63584B47C8 547 8 8 4 8 A3#B3"4C 8 53E 8 6 8 5 8 "6 E 8 547 8 8 A335B4 8 5478 354"B4 8 8 5 8 4B3747 8 54C 8 5 B374C 8 B 8 *567, 8 ( 8 A5A38 A348548B4CB4C8353 8585"85887BA"48B#858 B"A638B4 8

"B7BBCD 8 #B3 8 8 3 BC4B4 8 B# 8 "6 0 8 B"A8 8 547 8 547 8 #4C38 C63 8 A3#B3"7 8 4 8 A5 , 8 ( 8 "B7BBCD 8 54 8 B436 8 B 8 B 8 8 545D 8 B# 8 5 5 8 "6 8 A5D4C 8 BBE 8 6 8 5 8 8 663BA54 8 547 8 8 F654E 8 547 8 B 8 8 #4C3 8 C63 8 B43B 8 B# 8 B647 8 5 8 5 8 44 8 43#5 8 #B3 8

"6 588A3B4,8248"A"45B48B#858C43 8"B78#B38674C8B7D8 6A538 5# 5B48"B785AA7848"6 58C6388A347,88283B68 5# 5B48"B78#3B"858376 783544C8755E858485858"B78#B38 A558 5CC3C5B4 8 B#8 8 5# 5B48 36E 8 4 8 A3B7 8 58 B4#74 8

"563 8 B4 8 5 8 547 8 6A53 8 B 5B4 8 8 7BA7, 8 2 8 9:; 8 A8048 5# 5B485 635 D85478<=;8AB4 6586A538B 5B485 635 D885 78 448"6 58#4C38C6384858"0 B785478538A3#B3"7848#3B48B# 8 8 5"358547883B5B4853B6478 5"35858884B8BB8"AB354,

"D#32567$ 81547 8 ABE 8 5 8 "B7E 8 3547B" 8 7 B4 8 #B3E 8 B"A638 B4E8"5 485344CE87A8"5AE8#4C38"6 58C63

%888AF526F2A

1547 8 547 8 #4C3 8 C63 8 5 8 545D 8 A5D7 8 54 8 "AB354 8 3B 8 4 8 6"54 8 53 8 8A3B4, 8( 86"548 4B40B4 8478 8 8A3B48 B46 88 )1 858 B67 8 8 A337 8 547 8 354"7 8 4 8 8 B436B4 8 B# 8 8 >0(3563?8 353 8 A3B@ ,8 2 B374C 8B8 8B@ 8B# 8>0(3563? 85 84B 8 6"B758 16"5405 4 8 )43#5 8 #B3 8 8 53 8 8A3B4 8 547 8 "B3 8 A3 D 8 #B3 8 8 B4"AB353D8"6 8 B"ABB48B67887BA7E843845635854785478#4C3 8 C638A3#B3"78B6788"5AA7848358"8B8B647,8(38538C4# 548 54C8B857738688#38BB68B488B8A3 D8 5A63854783 BC4A 8

#4C38C63,8(85848588538B#8"54D8A3B68353 8B6C8B#48 835638#B 68B488"B8358 58438854788457848853848

#3B48B#88 5"358488BA40A5"8ABB4,8)48853 848B485854884C8

"B78"5D885AA78B8"B38 B"A885478AB8"AD4C858AB868#B38B63 8

#663 8 "6 0 8 #4C3 8 C63 8 3 BC4B4, 8 ( 8 67D 8 547 8 8 "B7 8 A3B76 78

B678 B4368B88545D8547874# 5B48B#88 535 3 8B#87##348

A54 8 BB82F65485478663BA54385AA7848"6 8A5D4C,

&88BF4FD828FD85F

F 4 8 353 8 474 8 B4 8 54 8 4 354C 8 43 8 4 8 74# 5B4 8 5478 3 BC4B48B#8C63848868B#87##348DA8B#8"BB48 5A638 4BBCB8 C38"BB484B3057E8533057E85478"53308 4BBC,

D53B68DA8B#8438"BB484B38EFGE=GEHG8B38 B""3 5843#5 E86 8 588C8@BD 8EIGE8B388)01<:84358"BB48 5A63868#3B"824"5ABBE 8 548A3B78350"85 8B8"BB484#B3"5B4,8*65DE 8D8538678#B388 3 BC4B48B#8C638A3#B3"7848A5 8B38B4854C8B@ 854788A3B7858 3B5B483A345B48B#88"BB4,

533057 8 D" 8 53 8 57 8 B4 8 BA 50"533 8 4BBCDE 8 6 8 5 8 D B48 '58B387A35 ,8)48E1GEG8#B38454 E88D B48D"8458678B8 5A6388

"BB4 8 B# 8 B4 8 A5D3, 8 ( 8 5" 8 B# 8 8 353 8 45 8 8 "B74C 8 B# 8 "6 8 A3#B3"54 8 D 8 64735474C 8 7##34 8 B44C 8 35C 8 4 8 B4 8 A5D4C, 8 )48 54B38 5E8353 38378B8575A88"B78B48A54B8A5D38EJGE84 85 8 B43678B8548B##048545D,8

53308D"87B84B83%6386@ 8B84538A 58%6A"48#B3835 4C8 5478538665D8578B48A58 B"A638B485AA3B5 ,8)48E9GE83 BC4B48B#8 8"6 58## 8B#88C653K8#4C38"BB48B487 38"8488A3BAB7E8 64C 8 5 8 #4C3 8 C63 8 3 BC4B4 8 57 8 B4 8 8 >6DC? 8 B"A63 8 B4 8 4A5#B3", 8 (8 5AA3B5 8 544B 8 5D8 85AA7 8 4 8 8A3#B3"54 , 8 )4 88 68A3B@ 8E<GE8"BB48B#88A54K854788678B8>#BB488"6 8 B3?E 8 4 8 "54 8 B 8 D4 3B4A 8 8 A3#B3"54 8 B 8 8 "6 8 B3, 8 2 8 5"35 8 458

"B647 8 5B 8 8 A54B 8 DE 8 B43E 8 8 547 8 B# 8 8 "6 54 8 8 CB5D8 545DA7E84B84858#4C38E85478 B4%64D8#4C38C6385384B83 BC4A7, 8 24B3 8 "B7BBCD 8 #B3 8 B"A8 8 #4C3 8 "6 5 8 C63 8 3 BC4B4 8 5 8 48

"A"47 8 4 8 8 '54735 8 D"E 8 4 8 8 57 8 B4 8 "5330 8 B"A638 B48"5C8545D8 4%68B87 8547874#D88#4C3A854788 43B78 B#885478B4858FL87B8E1:G,

2854884B48B#8E1:GE848A348488B3853 8585478 5# 5B48"B78 5 8 A3" 8 8 3 BC4B4MB 5B4 8 B# 8 7##34 8 547 8 6A53 8 4 8 8 64C8

"6 08#4C38C63848868B#8584C8"0B#0#C87A8 5"3582'9 8 5"LB5378554B3,

3 Hand Skeletal Model

3.1 Overview

358"B7888B8338B7D8B3854786A538ABB4838#3B"8FL8B38

#3B"87A8"5A,8(8 B""B4D8678#35"4B388B8#858B48"B78B8588

"5 8B358#56385478B85AAD84384"5 8B83#488B48

E1F01=G, 8 2 8 "5@B3 8 3#34 8 4 8 8 #7 8 8 8 N4 8 B7D 8 335 8 5CB3"8

7 378D8BB4885,8E11G8438F547B"89 B48B382F93853835478B 8

A3#B3"8A8048B7D8 5# 5B4,8(85AA3B5 85848A3B483B68B6C8

45D833 78B8438B7D8C638768B84B38"5B4,8!5DE8N488

5,8E1G84C57885"85AA3B5 88"8B4885478B48547888

3D 8 A3B"4C 8 A3#B3"54 , 8 E 8 8 8A3"45B4 8 45 8 "7 8 B 8 5 8 A3BB#0B#0

B4 A8583 BC4A82"3 548C48!54C65C87C,86 85AA 5B488#538#3B"8 B63868 584 8854788665D8#5 4C88 5"358488 5 58BA40A5" 8 ABB4854788"654B68#4C38C6383 BC4B48584B8485657,8

(B6C85B873578#3B"8BB4885,8E11GE8B6387BA"484C588 B"A8868 584388547888 64C8"6 08#4C38C63858538B#8 C38B3738B#8 B"A8D,828#3E84868583085478"B78481858 584 B"A5885478582A5"854784338584858#4C385478#4C3A85 8 7A 7848C,81,8C88588"B788 B"A884B6C8B8545D8#485478 B4#C635B4854785885"8"8"A3854881<858"B7867848E1GE8 6 8 A3B74C 8 8 5# 5B4 8 33B3, 8 C 8 1, 8 B4 8 A804 8 5# 5B48 B547864C887 678"B7,

!%!815478"B7867848186A53E87A8"5AE8 B33AB474C8C3B64703685478A80 48 5# 5B4,

3.2 Training with Random Decision Forests (RDF)

286CC7848E1101GE8483548F547B"89 B48B382F938B858B8A3#B3"8 A8048 5# 5B4,8F98E1=G88548"A3B"48B#889 B48(38"5 48 5344C85AA3B5 843858 B"A88A3B"88A848"A87 B48B85858 538B#487A 785884B78B#8583828584C88#4587 B43, 8B385 8 38B#8548F985868B#8A8812 #3B"8"5C8B#883544C875558538678B8 354883,8(B8"88A3B 4C85478""B3D8 BE848686A8B8J87 B4838 4858"58"587A8B#8:,8(4E8#B385 84B7E8483547B"D8C4358:::8458 5#385478#B385 8B#88=:8 5477583B7,8(8458 5#384868 2,,8#56338 B"A53887A8B##84858A # 84CB63854788"538B8 E11G8 588458A3B78B885 63585478#584B6C,8563858A88x8B#87A8"5C8388 68 B"A67858587##34 8B#87A88#B38B## 8u 547 v normalised w.r.t.

depth858x,85888A347848%65B481,

4

23E 13 = 32 1 + 5

32 13 3 − 321 + 6

32 13 3 213

C84 8"65 87558A53B485 884B7 8D83B74C88#563 83AB4 8

64C85 8 5477583B785478 B"A68843BAD86%648B88A53B48B#8

8755,845DE848A88 B"45B48B#8#563854783B7858"58"A8 84#B3"5B48C54E84 874B887##34 8448 633484B784#B3"5B48 43BAD8547886"8B#843BAD8B#886838364C8#3B"887558A53B4,89558 A53B485478458 5#38 B4853848 B"A6783 63D8B488#85478 3C 8 63 8 4 8 5 8 A3#8 8 B373E 8 64 8 3 8 "58"6" 8 7A 8 8 35 7 8 B3 8 8 4#B3"5B48C548CB86473858#8783B7,8C8465D8B384838588 A3B5D8736B48B#887##34854786A53,8(B8736B48 548830 B"A6785#34537864C8588"5C8#3B"883544C8755584 E85 B374C8 B8E1GE85B48CD8C383 BC4B4835,

)4 8 B63 8 5E 8 8 3544C 8 5C 8 8 A3#B3"7 8 6AB4 8 =:: 8 547057 8 "5C8 3 B377848B6387A8 5"35,8(875558C5384A68#3B"8=87##348A3B48

"654C8"6 0854785478#4C3 8C63 848#3B4 8B#88 5"35 86 8588

>53ACCB?C63E8587A 7848C,,

!8&!8248>53ACCB?8 B385478A54B08C638"65B4827A8"5A85478C3B6470363

C84B85E8#8E11G85478E1G86CC8868B#8647378B65478"5C8#B38 85"48B#85858"B78358B8583B578A 36"8B#874 8B7D8B38 547 8AB63E 8 4 8 8 5E8 48 B638 33 7 86 5E 8 8 8AB 8B8 5 8 5#D4C8 5# 5B483684858376 785"A,

3.3 Pixel-wise classification

B3 8 5 8 A8 8 B# 8 54 8 4A6 8 7A 8 "5CE 8 8 3547 8 7 B4 8 3 8 47A474D8 B6A6858A3B5D8736B48588358B8854786A5385C4"4,8(8 736B4853848535C78B383D8385"B4C887 B48#B385478#B3"8 8#458A88A3B5D8B8B4C8B85 86A538B#88"B7,8

5B4E8#3B"85 86A538A3B5D8"5A848"5886A538ABB4864C88 548#85CB3"8E1JG84 858857545C8B#8 B43C4C83D8%6 D,8)85B8 5B4868B8#3884B8B488 5# 5B48"563"48B4858A888488 3B74C8B#88A3B58#B38 B"A64C8874D8"5B3,

3.4 Performance

(B8"56388A3#B3"54 8B#8B638A3BAB78"B784858 B"A53788B6A68B#8

B63 8 A804 8 5# 5B4 8 4 8 1= 8 "5C 8 21H=81:A83 8 5 8 4 8 58

A3B6D8547057,8)48 B4357 B48B8E11G85478E1G8487784B868D48

"5C858884B8#58B83A3B76 8885 88A38C63,814 E885470 54C8"5D886@ 8B8B"8535B4 8#885"8A3B 845 85 B"A7 8 358",

C,8J,58B488A3#B3"54 8#B385 8B#8B638"B786A53848548535C8 5# 5B48358B#89:;, 8154786A538A3#B3"54 8 B33AB474C8B843E88

#4C3E834CE8"77E847885848586"8A885B88535C85 635 DE84 8 8CD85B89:;,81B43E8B3854786A5386 8586"8B38A5"8538

"B38D8B88B 6778D8B38#4C384885478 B4#C635B4848677,8 5B4B3DE88#4C3A88858B438 5# 5B482H:09:;383584 8 B6788 8A5478D88#5 85838"5C78A88"53868"AD4C8#4383544C8 75582A 5DE858B4D8A885383547B"D85"A784883544C8A3B 3,86 8 56"AB48 B67883#78D8303544C8B638"B7848"B38"5C,

!8'!8'3#B3"54 8B#8B638"B7,82538'88 5# 5B4835,8238235C8A88754 8#3B"8 8@B48 43B78"578B88@B48 43B78C3B647036,

277B45DE 8 B 8 "563 8 8 A3 B4 8 B# 8 B63 8 @B4 8 "5B4 8 4 8 B"A53 8 8 535C 8 754 8B# 8 8 @B4 8337 8 #3B" 8 854 8 # 85CB3" 85C54 8B63 8 C3B6470368755,83B"8B6381=88"5CE88A3#B3"54 83AB37848C,8J,,8 B485858@B48ABB4888548J8A88#3B"8B638C3B64703688 A88 A5"8@B484 8885 635,82C54E8888A5478D88#5 8588A5"8 8 B#48"5C7848358A 8768B8#4C38B 6B48"54C885 6358 43B7 8

"5B48"6 8"B38 B"A8,

B8#53E88A3#B3"54 8"86## 48B886784858#4C38C6383 BC4B48 AA4, 8 1B43E 8 B 8 5B4 8 #4 8 AB 8 3 BC4B4 8 4 8 5 8 4#B3"5B4 8 5B6 8 8 B4#74 8 4 8 8 547 8 6A53 8 ABB4 8 "5B4, 8 74 8 B6B4 8 4 8 77 8 4B8

"A"48D84B6788B8 B"A5388A3B8"B"48 B38488B6A68#3B"88

"548#835B4E85884B678A3B78548434C8"5638B#8B48"AB35488 3378B 58"58"58B#88A3B5D8736B48,

(8882AE72A84A68D57DF)D7

( 8 A5A3 8 A347 8 54 8 "A"45B4 8 B# 8 5 8 C43 8 "B7 8 #B3 8 674C 8 B7D8 6A538 5# 5B48"B785AA78B8"6 58#4C38C638#B388A335B48B#8 8 )1, 8(8 "B7BBCD8 54 8 B436 8 B8 B 8 8 545D 8 B# 8 5 5 8 "6 8 A5D4C8 BB8685B8B88#4C38C638 B43B8B#8B6478585844843#5 8#B3 8

"6 588A3B4,89:;8A8048 5# 5B485 635 D85478<=;8AB4 6586A538

B 5B485 635 D85385 78448"6 58#4C38C6384858"0 B785478

538A3#B3"7848#3B48B#88 5"35,82858488A84884538#663E888A54478B8

A3#B3" 8 755 8 #6B4 8 B# 8 4B 8 7A 8 5"35 8 4 8 B373 8 B 8 5 8 3 8 36 8 5478 577388A3B"8B#8 485478#0B 6B4,

CA23ED6DDAF7! 8( 8 353 8 574C 8 B 8 8 36 8 5 8 3 7 8 #6474C 8 #3B" 8 8 663BA548*4B4E84835"4B38'3BC35""82'I0)(0:110<3864738C35485C3"484O8 H::HIH,

*DD5DAD7

1, F55""54545E5,E8L5 %65E8,B86##B30578545D8B#8B44C8"B"4B874 8B#8 54 A5B48## ,8(8PB63458B#85486 8F53 E8JI2F3B8JJ<8Q8J=182::<3

, 9"B6 3B4E 8 ,E 8 24#E 8 2,E 8 56$E 8 F,B 8 735B4 8 B4 8 B4 8 54C 8 4 8 B48 A3#B3"54 ,8)48'3B 74C8B#82 B6 E8PB63458B#882 B6 58B D8B#82"3 5E 8 B6"81JE8A5C8J1J821<<F3

J, '5"3E,8E8'#B3733E8',8R,B83B"8"D85478B8DB63853B88#5 8B#8"3848A3#B3"54 8 5478A3 AB4,8)48,8CBB7E8,8!6 E8F,8L3B 537E8,877B4S8P,82,8BB758267,38)4 8 '3B 74C8B#88H8)4345B458B4#34 8B486 8'3 AB485478BC4B4,NE 8

*NB8N8*43D82:::3

F, 2D4537E8F,E8!BE8,9,E8'5357BE8P,82,B828B"A5 E8C3E8C53584B3854B38

#B38)435 8954 864"E848'3B ,8B#8L58::HE8(8)6668)4345B458CB3BA 8 B48C53585478)"A5458LB7D84B3854B3E85"37CE855 6E8AA,8H=0 I:E82A38J0=82::H3

=, B76DE8(,E8143DE8,E8B4E82,B8(72(6F85478NF77596B81C0FB6B4854781C0 A78F50"84B38)43#5 E8)48'3B 74C8B#88)4345B458B4#34 8B48548 )43#5 8#B386 5868A3B4825)60:F3E815"5"56E8P5A5482::F3

H, (B7B3B##E(,B 8 C3 8 7C5M545BC 8 4B3 8 #B3 8 "6 8 547 8 754 8 A3#B3"54 , 8 )4 8 '3B ,8 5)68K11E8A5C8=1=Q=19E87BE85B345D82:113

I, 3643CE89,B8638F BC4B48#B38B476 4C8B"A6386 ,8F378P54653D81:E8 ::<E8#3B"B8ABMM 63, ,738,76M353 MC63F BC4B482::93

9, L634E2,8,E 8C5473DE,B 8D658B78#B388F358B#865384C34C, 8)48 '3B 74C 8 B# 8 8 )4345B45 8 B4#34 8 B4 8 54 8 )43#5 8 #B3 8 6 5 8 68A3B4, 8 '53E8354 82::H3

<, (868A3B@ E8583378B48=

8P648:1J8B8 ABMM76D,"6 ,6 , 5M"6M"6,"

1:,5453E 8,E 8(5C53E 8 2,E8 9"3BAB6BE 8N,E 85453E 8 82,B8 8 4C38 "6 58 C638 3 BC4B4848J98A5 84B6854D854C8436"48#B38A3#B3"4C853E 8)4345B458 PB63458B#82385478( 4BBCD8248A33

11,BB4E8P,8E853AE8(,E8NA"54E82,E8ACB4E82,E84B BE8,E8L5E82,E8BBE8,E 8 BB3E 8 F,B 8 >F50" 8 6"54 8 AB 8 3 BC4B4 8 4 8 A53 8 #3B" 8 4C 8 7A 8 "5CE? 8 B""64 5B48B#882E8B,8=HE84B,81E8A,811H82:1J3

1,1,8N4E8,E8NT35UE8,E8N535E8V,E825364E8!,B8>F58"85478AB8"5B4864C87A 8 4B3E?E89A85"358#B38B"A6382:1J38

1J,B"54 6E 8 9,E3E 8',B8 >548 #B 858 3B685AA3B5 8B4537 8 #563 8A5 8545DE? 8 3777289ABCADEFBC2B2AEE9B2BACFC2AB2ADFB23BEFBDE8B,8FE84B,8=E8AA,8H:JQ H1<82::3

1F, 453AE 8 !, 8 2,E 853D54E 8 2,E 856E8 9,E 8555E 8 5,B 816"54 8B48 35 4C 8#3B"8 7A8755864C8CB7 8754 85478BA 58#B4,8)"5C85478DB48B"A64CE8B,8J:E8 4B8JE8A,81I0H82:13

1=,7B4B"7E 8 ),E 8 ND35AE 8 5,E 8 23CD3BE 8 2,B 86## 4 8 "B7057 8 J7 8 35 4C 8 B# 8 5478

53 65B4864C84 ,8)48B8L385 48DB48B4#34 ,8A,81:1,101:1,11882:113

1H,L3"54E8!,B8>F547B"8#B3E?85 485344C82::13