On-line fusion of trackers for single-object tracking

(1)

HAL Id: hal-01635420

https://hal.archives-ouvertes.fr/hal-01635420

Submitted on 24 Apr 2019

HAL is a multi-disciplinary open access archive for the deposit and dissemination of sci- entific research documents, whether they are pub- lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

On-line fusion of trackers for single-object tracking

Isabelle Leang, Stéphane Herbin, Benoît Girard, Jacques Droulez

To cite this version:

Isabelle Leang, Stéphane Herbin, Benoît Girard, Jacques Droulez. On-line fusion of trackers for single- object tracking. Pattern Recognition, Elsevier, 2018, 74, pp.459-473. �10.1016/j.patcog.2017.09.026�.

�hal-01635420�

(2)

*Manuscript

Click here to view linked References

(3)

(4)

(5)

(x, y, w, h)

(x, y) (w, h)

(6)

(7)

(8)

(9)

(10)

(11)

n k ?

4 k

3 ?

4 k ?

2 k

n k

(12)

B B

⁰

IoU(B, B

⁰

) = | B \ B

⁰

|

| B [ B

⁰

|

(13)

(14)

#170/271

NCC KLT CT STRUCK DPM MS

#174/271

NCC KLT CT

DPM

#176/271

CT

DPM

#8/145

#70/145

#75/145 NCC

DPM

#50/326

#84/326

#90/326

STRUCK

MS

#110/366 KLT CT STRUCK DPM MS

#145/366 KLT CT STRUCK MS

#150/366 KLT

MS

#55/164

NCC KLT CT STRUCK DPM

#73/164

#79/164

KLT CT STRUCK DPM

#160/178

#173/178 NCC

CT STRUCK MS

#177/178

(15)

50 100 150 200 250 300 350 ASMS

MS DSST DPM STRUCK CT KLT NCC

bolt

time

50 100 150 200

ASMS MS DSST DPM STRUCK CT KLT NCC

gymnastics

time

50 100 150 200 250 300 350 ASMS

MS DSST DPM STRUCK CT KLT NCC

handball1

time

t d

ⁱ_t

T

i

i 2 [1, M ]

I M

N

I = X

N t=1

Y

M i=1

d

ⁱ_t

(16)

#2/271

x

y

10 20 30 40 50 60 70

5 10 15 20 25 30 35 40 45

−0.2 0 0.2 0.4 0.6 0.8

x

y

1020304050607080

10

20

30

40

50

60

70

80 0

50 100 150 200 250

x

y

10 20 30 40 50 60

10

20

30

40

50

60 −1

−0.8

−0.6

−0.4

−0.2 0 0.2 0.4 0.6 0.8 1

x

y

51015

5 10 15 20 25 30 35 40 45

−240

−220

−200

−180

−160

−140

−120

−100

−80

−60

x

y

51015

5 10 15 20 25 30 35 40 45 50

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4

M T = { T

₁

, T

₂

, ...T

_M

}

)

s

ⁱ_t

= 0 s

ⁱ_t

_M

}

B

₀

I

₀

I

_t

T

_i

i 2 [1, M ]

B ˆ

ⁱ_t

B ˆ

_t

=

( ˆ B

_t¹

, B ˆ

_t²

, ... B ˆ

_t^M

) B ˆ

_t^{f usion}

t T

_i

B ˆ

_tⁱ

it

T

_i

i 2 [1, M ]

(20)

Tracker 1

1. Tracker Parallel Running

𝐵𝐵�_𝑡𝑡²,𝜙𝜙_𝑡𝑡² 𝑩𝑩�_𝒕𝒕,𝝓𝝓_𝒕𝒕 𝐵𝐵�_𝑡𝑡¹,𝜙𝜙_𝑡𝑡¹

𝑩𝑩�_𝒕𝒕,𝒔𝒔_𝒕𝒕 Tracker 2

Tracker M 𝐵𝐵�_𝑡𝑡^𝑀𝑀,𝜙𝜙_𝑡𝑡^𝑀𝑀

𝑠𝑠_𝑡𝑡¹ Drift Predictor 1

Drift Predictor 2

Drift Predictor M 𝑠𝑠_𝑡𝑡²

𝑠𝑠_𝑡𝑡^𝑀𝑀

Fusion

𝑩𝑩�_𝒕𝒕𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒕𝒕𝒄𝒄𝒄𝒄,𝒔𝒔_𝒕𝒕

𝐵𝐵�_𝑡𝑡𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓 𝐼𝐼_𝑡𝑡

2. Tracker Selection

3. Tracker Fusion

4. Tracker Correction System

It Ti i2[1, M]

Bˆtⁱ i

t

Bˆt = ( ˆB_t¹, ...Bˆ_t^M) t= ( ¹_t, ... ^M_t )

sⁱt ={0,1} st =

(s¹_t, ...s^M_t )

Bˆ_t^{f usion}

Bˆi,corrected

t = ˆB^{f usion}_t Bˆ^corrected_t = ( ˆB1,corrected

t , ...BˆM,corrected

t )

(21)

DP

_i

s

ⁱ_t

t

s

t

= (s

¹_t

, s

²_t

, ...s

^M_t

)

B

_t

IoU ( ˆ B

ⁱ_t

, B

t

) < 0.2 DP

i

s

ⁱ_t

= 0

B ˆ

^{f usion}_t

K K  M T

_i

s

ⁱ_t

= 1 B ˆ

^{f usion}_t

K K

d

_ij

i j w

_i

i

w

i

=

P 1

j6=idij

P

K k=1 P 1

j6=kdkj

B ˆ

_t^{f usion}

=

B ˆ

_t^{f usion}₁

B ˆ

i,corrected

t

, i 2 [1, M ] o

B ˆ

i,corrected t

B ˆ

_t^{f usion}

B ˆ

_tⁱ

B ˆ

^corrected_t

(23)

(24)

(25)

(w, h)

dx

_max

> 200 t

d

area

= (area(t) area(t 1))/area(t 1) area(t)

thr(t) = max 0.1 ⇤ (max min) max

min t

varxy

_10max

= p

varx

_10max²

+ vary

_10max²

/min(w, h) dxy

_max

= p dx

max2

+ dy

max2

/min(w, h) varxy

10max

t dxy

_max

t t 1

P SR

f bratio = (max

_F

µ

_B

)/

_B

max

_F

µ

_B _B

(27)

3 ⇤ (w, h)

dbest = score(best1) score(best2)

obest = IoU (best1, best2) dbest best1

best2 best1

> 0.3 best2

best1

bhatta

_f

bhatta

_b

(28)

(29)

no DP Ideal DP BF BC BI BI+BF BI+BC 0

50 100 150 200 250 300

Average robustness per selection method

Total VOT2015 VOT-TIR2015 VOT2013+

Selection method

Robustness (number of drifts)

P UD UA RD

0 50 100 150 200 250 300

Average robustness per correction method

Correction method

Robustness (number of drifts)

(30)

(31)

± ± ± ±

± ± ± ± ± ±

± ± ± ± ± ± ±

± ± ± ± ± ±

(32)

0 100 200 300 400 500 600 100

150 200 250 300 350 400

Incompleteness

Fusion performance (nb drifts)

VOT2015

2 trackers 3 trackers 4 trackers

0 100 200 300 400 500 600

100 150 200 250 300 350 400

Incompleteness

Fusion performance (nb drifts)

VOT2015 : 2 trackers

NCC−KLT NCC−CT NCC−STRUCK NCC−DPM NCC−DSST NCC−MS NCC−ASMS KLT−CT KLT−STRUCK KLT−DPM KLT−DSST KLT−MS KLT−ASMS CT−STRUCK CT−DPM CT−DSST CT−MS CT−ASMS STRUCK−DPM STRUCK−DSST STRUCK−MS STRUCK−ASMS DPM−DSST DPM−MS DPM−ASMS DSST−MS DSST−ASMS MS−ASMS

(33)

Gain

Indiv

(34)

(35)

(36)

(37)

(38)

(39)

> >

(40)

(41)

(42)

(43)

(44)

(45)

(46)

(47)

(48)

(49)

Isabelle Leang graduated from the Ecole Nationale Supérieure de l’Electronique et de ses Applications (France), received a Master degree in Computer Sciences from the Université de Cergy-Pontoise (France) in 2012. She is actually preparing a Ph.D. degree in the Information Processing and Modeling Departement at ONERA (France).

Stéphane Herbinreceived an engineering degree from the Ecole Supérieure d’Electricité (Supélec), the M.Sc. degree in Electrical Engineering from the University of Illinois at Urbana-Champaign, and the Ph.D. degree in applied mathemat- ics from the Ecole Normale Supérieure de Cachan. Employed by ONERA since 2000, he works in the Information Processing and Modeling Department. His main research interests are stochastic modeling and analysis for object recognition and scene interpretation in images and videos.

Benoît Girard received a Ph.D. degree in Computer Science (2003) from the Université Pierre et Marie Curie (Paris, France). He currently work as a Research Director at the Centre National de la Recherche Scientifique. His main research interests are action selection, reinforcement learning and decision making in animals and robots.

Jacques Droulezreceived a mathematical training at Ecole Polytechnique (Paris, France) and a MD (1982) from the Uni- versity Paris 6. He is currently Research Director at the Centre National de la Recherche Scientifique. His main research interests are motion and object perception, sensori-motor control and Bayesian modeling of biological systems.

1

*Author Biography

(50)

P UD UA RD 0

50 100 150 200 250 300

Average robustness per correction method

Correction method

Robust ness (numbe r of drif ts)

correctionstep.pdf

(51)

#110/366 KLT CT

STRUCK DPM MS

fish1_img_0110.eps

(52)

#160/178

NCC KLT CT

STRUCK DPM

MS

quadrocopter_img_0160.eps

(53)

50 100 150 200 250 300 350 ASMS

MS DSST DPM STRUCK CT KLT NCC

bolt

time

tracklets_vot2013_bolt.eps

(54)

#150/366 KLT

MS

(55)

x

y

5 10 15

5

10

15

20

25

30

35

40

45

50 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4

map_dsst_0002.eps

(56)

x

y

10 20 30 40 50 60 70

5 10 15 20 25 30 35 40 45

−0.2 0 0.2 0.4 0.6 0.8

map_ncc_0002.eps

(57)

#145/366 KLT CT

STRUCK MS

(58)

0 100 200 300 400 500 600 100

150 200 250 300 350 400

Incompleteness

Fusion performance (nb drifts)

VOT2015

2 trackers 3 trackers 4 trackers

vot2015_fusion99.eps

(59)

#50/326

NCC KLT CT

STRUCK DPM

MS

dinosaur_img_0050.eps

(60)

#173/178 NCC

CT

STRUCK MS

(61)

#174/271

NCC KLT CT DPM

bicycle_img_0174.eps

(62)

0 100 200 300 400 500 600 100

150 200 250 300 350 400

Incompleteness

Fusion performance (nb drifts)

VOT2015 : 2 trackers

NCC−KLT

NCC−CT

NCC−STRUCK

NCC−DPM

NCC−DSST

NCC−MS

NCC−ASMS

KLT−CT

KLT−STRUCK

KLT−DPM

KLT−DSST

KLT−MS

KLT−ASMS

CT−STRUCK

CT−DPM

CT−DSST

CT−MS

CT−ASMS

STRUCK−DPM

STRUCK−DSST

STRUCK−MS

STRUCK−ASMS

DPM−DSST

DPM−MS

DPM−ASMS

DSST−MS

DSST−ASMS

MS−ASMS

vot2015_2trackers.eps

(63)

50 100 150 200 ASMS

MS DSST DPM STRUCK CT KLT NCC

gymnastics

time

tracklets_vot2013_gymnastics.eps

(64)

#55/164

NCC KLT CT

STRUCK DPM

motocross1_img_0055.eps

(65)

#73/164

NCC KLT CT

STRUCK DPM

MS

(66)

#79/164

KLT CT

STRUCK DPM

(67)

x

y

5 10 15

5

10

15

20

25

30

35

40

45 −240

−220

−200

−180

−160

−140

−120

−100

−80

−60

map_ct_0002.eps

(68)

#84/326

NCC KLT CT

STRUCK DPM

MS

(69)

Tracker 1

1. Tracker Parallel Running

Tracker 2

Fusion

𝑩𝑩 �

_𝒕𝒕𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒕𝒕𝒄𝒄𝒄𝒄

, 𝒔𝒔

_𝒕𝒕

𝐵𝐵�

_𝑡𝑡𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓

𝐼𝐼

_𝑡𝑡

2. Tracker Selection

3. Tracker Fusion

4. Tracker Correction

System

schemablock-crop.pdf

(70)

#177/178

(71)

#2/271

map_img_0002.eps

(72)

50 100 150 200 250 300 350 ASMS

MS DSST DPM STRUCK CT KLT NCC

handball1

time

tracklets_vot2015_handball1.eps

(73)

x

y

10 20 30 40 50 60 70 80

10

20

30

40

50

60

70 80 0

50 100 150 200 250

map_klt_0002.eps

(74)

#170/271

NCC KLT CT

STRUCK DPM

MS

(75)

#176/271

CT DPM

(76)

#70/145

NCC KLT CT

STRUCK DPM

MS

gopr0020_moto_img_0070.eps

(77)

x

y

10 20 30 40 50 60

10

20

30

40

50 60 −1

−0.8

−0.6

−0.4

−0.2 0 0.2 0.4 0.6 0.8 1

map_struck_0002.eps

(78)

#8/145

NCC KLT CT

STRUCK DPM

MS

(79)

#75/145 NCC

DPM

(80)

#90/326

STRUCK MS

(81)

no DP Ideal DP BF BC BI BI+BF BI+BC 0

50 100 150 200 250 300

Average robustness per selection method

Selection method Robust ness (numbe r of drif ts)

selectionstep.pdf

(82)

Figure

Click here to download high resolution image

(83)

Figure

(84)

Figure

(85)

Figure

(86)

Figure

(87)

Figure

(88)

Figure

(89)

Figure

(90)

Figure

(91)

Figure

(92)

Figure

(93)

Figure