HAL Id: hal-00935612
https://hal.archives-ouvertes.fr/hal-00935612
Submitted on 23 Jan 2014
HAL is a multi-disciplinary open access
archive for the deposit and dissemination of
sci-entific research documents, whether they are
pub-L’archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
Software description and configuration
Christine Guillemot, Laurent Guillo, Marco Cagnazzo, Giuseppe Valenzise,
Béatrice Pesquet-Popescu
To cite this version:
Christine Guillemot, Laurent Guillo, Marco Cagnazzo, Giuseppe Valenzise, Béatrice Pesquet-Popescu.
Software description and configuration. 2013, pp.11. �hal-00935612�
S hémas Per eptuels et Codage vidéo 2D et 3D ANR-09-BLAN-0170
LivrableD5.3 26/09/2013
Software des ription and onguration
Christine GUILLEMOT IRISA
Laurent GUILLO IRISA
Mar o CAGNAZZO LTCI Giuseppe VALENZISE LTCI Béatri e PESQUET-POPESCU LTCI
Contents
1 DCR 4
1.1 Overview . . . .. . . .. . . .. . . .. . . .. . . 4
1.2 Matlabfun tions . . .. . . .. . . .. . . .. . . .. . . 4
1.3 Modieden oder len ode.exe . . . .. . . .. . . .. . . .. . . 5
2 WTM 6 2.1 Qui koverview . . . .. . . .. . . .. . . .. . . .. . . 6
2.2 Someimplementationdetails . . . . .. . . .. . . .. . . .. . . 7
2.2.1 WTMa tivation . . . .. . . .. . . .. . . .. . . 7
2.2.2 Templateshapesignalling . . .. . . .. . . .. . . .. . . 8
2.2.3 Numberanddimensionsofsear hwindows .. . . .. . . .. . . 9
2.3 Conguration . . . . .. . . .. . . .. . . .. . . .. . . 9
2.4 Exampleofuse . . . .. . . .. . . .. . . .. . . .. . . 9
Introdu tion
Thisdo umentgathersinformationabout ongurationandinstru tionsforuseoftwo software arriedoutduringthePerseeproje t: DCRandWTM.
1 DCR
The Don'tCare Region (DCR) approa h is basedon theideathat in multiple-view-plus-depth video, depth maps are not dire tly viewed, but are only used to provide geometri information for viewsynthesis at de oder. Thus, aslong asthe resulting geometri error doesnotleadto una eptable quality forthe synthesized view, ea h depth pixelonlyneedsto bere onstru ted atthede oder oarsely withinatolerable range. WerstformalizethenotionoftolerablerangeperdepthpixelasDonâtCare Region(DCR),bystudyingthesynthesizedviewdistortionsensitivitytothepixelvalue asensitivedepth pixelwillhaveanarrowDCR,andvi eversa.
1.1 Overview
Wenowdeneper-pixelDCRsfordepthmap
D
n
,assumingtargetsynthesizedviewisn
. Inthefollowing,wewillratherrefertothedisparityeldd
n
,whi h anbeobtained fromthedepthon ethe ameraparametersareknown. Apixelv
n
(i, j)
intexturemapv
n
, withasso iated disparityvalued
n
(i, j)
, anbemapped to a orrespondingpixel in viewn
+ 1
throughaview synthesisfun tions(i, j; d
n
(i, j))
. Inthe simplest ase where the views are aptured by purely horizontallyshifted ameras,s(i, j; d
n
(i, j))
orrespondsto apixelin texturemapv
n+1
ofviewn
+ 1
displa edinthex
-dire tion by an amountproportional tod
n
(i, j)
. The viewsynthesis error,ε(i, j; d)
, an thus bedenedastheabsoluteerrorbetweenre onstru tedandoriginalpixelvalue,given disparityd
for pixel(i, j)
; i.e.,ε
(i, j; d) = |s(i, j; d) − v
n
(i, j)| .
Ifd
n
is ompressed, there onstru teddisparityvalue
d
˜
n
(i, j)
employedforviewsynthesismaydierfromd
n
(i, j)
byanamounte
(i, j) = ˜
d
n
(i, j) − d
n
(i, j)
,resultingina(generallylarger)view synthesis errorε(i, j; d
n
(i, j) + e(i, j)) > ε(i, j; d
n
(i, j))
. We dene the Don't Care RegionDCR(i, j) = [DCR
low
(i, j), DCR
up
(i, j)]
as the largest ontiguousinterval of disparity values ontaining the ground-truth disparityd
n
(i, j)
, su h that the view synthesis errorfor anypoint of the interval is smallerthanε(i, j; d
n
(i, j)) + τ
, for a given thresholdτ >
0
. Note that DCR intervals are dened per pixel, thus giving pre ise informationabouthowmu herror anbetoleratedinthedisparitymaps.TheDCRinformation anbethenusedinordertoperformamoreee tivemotion estimation, to en ode the predi tion residual, and to enhan e the use of the SKIP mode. Wehaveimplementedin Matlabafun tion that angeneratetheDCR's ofa MVDsequen e,andsavethemintoabinarylethat aninturnbeusedbyamodied H.264/MPEG-4AVCen oder(JMreferen esoftwarev. 18.0)
1.2 Matlab fun tions
[DCR_low DCR_up℄ = generate_DCR(isLeft, thres, depth_s ale)
Inputs
isLeft: binaryagindi atingwhethertheDCTistobe omputedfortheleft orfor therigthview. E.g., ifyouhaveviews 3and5in Kendo,andyou haveto synthesize view4, you shall use isLeft =1 to generate the DCR
neededtoen odedisparityofview3,and isLeft=0togeneratetheDCR relativetodisparityofview5. Notethat DCRisdenedastheworst- ase errrorwhen youhavetextureand depthfrom onesameview(e.g. texture anddepthfromview3)andyouwanttogeneratetheotherview(view5)
thres: itisthe
τ
in thereport (andinourPCS paper): thehighesttolerated thresholdto denetheDCR. Weadvi etouseτ
= 5
orsimilarvalues depth_s ale: itisthes alingfa torfromdepthtodisparity. Itmustbeknownapriori. It depends onthe sequen e,that should in any asebere tied. ForKendo,thevalueto useis0.204
Outputs
DCR_lowandDCR_up,respe tivelythelowerandupperboundofDCRinterval perpixel. Inorderto beusedbytheen oder,theymustbe onvertedinto abinarylebyusingwrite_DCR_bin.m
Note that this fun tion generates aDCR for asingle image. So you need to allit image-by-imageforanentiresequen e. Theinputsofthes riptsarethetextureand the depth of the right and left view, that should bepassed aspng les with names (left_ depth.png, left_texture.png,et .)
write_DCR_bin.m
Thiss riptwritesamatrix(
M
× N × F
,withM
× N
isthespatialresolutionandF
thenumberofframes)intoalethat anbeusedbytheen oder. TheDCRmustbe stored into variables named DCR_lowand DCR_up; theywill be written into two les named DCR_low.binand DCR_up.binPleasenote thatgenerate_DCRonlygenerates one2Dmatrix. Itisuptotheusertogeneratedasmanymatri esasyouneedandto sta kthemintoa3Dmatrix.1.3 Modied en oder len ode.exe
This is the modied version of H.264 (JM v.18.0) using the les DCR_low.bin and DCR_up.binprodu edbythewrite_DCR_bin.mfun tioninordertoperformthe DCR-baseden odingasdes ribedinthedelivrableD4.3andin thePCSpaper. Itworksin baselinemode,withsomerestri tions,asone anseefromtheen oder_baseline. fg le. In parti ular, theRDO should behigh omplexity and themotionestimation should befullsear h.
2 WTM
WTMisanintrapredi tionmethodbasedonalinear ombinationoftemplate mat h-ingpredi tors. Themethodwaspreviouslydes ribedin[1℄. Afteraqui kreminder,the followingse tionspresentshowsomedetailspe uliartothismethodwereimplemented and howto ongureWTM.Anexampleofuseisthengiven.
2.1 Qui k overview
WTM aimsat providing anintra predi tionforblo ksof
4
x4
,8
x8
,16
x16
and32
x32
sizes. Thispredi tionisbasedonalinear ombinationoftemplatemat hingpredi tors belongingtothe ausalneighbourhood.Figure1:Sear hregionsfrom ausalneighbourhood.
Then,
N
blo ksB
i
surrounded by the best mat hing areas are used to ompute predi torsP
i
,whi harethenaveragedtogetthepredi tionP
oftheblo kB
:P
=
1
N
N
X
i=1
P
i
(1)WTMrelies onthisgeneralapproa hbut therearethreemainenhan ements:
•
it uses4
dierent template shapes whateverthe blo k size: the traditional L-shapewhi his1pixellargeandthreeothershapeswiththeleft,thetoppartof both anbe4pixellarge. However,onlyonetemplateshapeisusedtodetermine alltemplatepredi tors.•
the orrelationfa torsisbasedonthedotprodu tbetweenthetemplateandthe templatepredi tors.•
template predi tors are not sear hed within all the ausal neighbourhood but within only two or three sear h windows. The number of sear h windows isrelatedtotherankoftheblo ktobepredi tedwithinthepredi tionunit(PU) andtheirsizedepends onthesizeoftheblo ktobepredi ted.
Formoredetailsaboutthesethree hara teristi ssee[1℄.Thefollowingse tionsgives informationabouthowtheyhavebeenusedandimplemented.
Figure2:Shapeoftemplates.
Figure3:Sear hwindowspositionsrelativelytoblo kB.
2.2 Some implementationdetails
Distin tive features listed in the previous se tions lead to the following hoi es of implementation.
2.2.1 WTM a tivation
WTM is notalways a tivated forall PU sizes. Is a tivation depends onthe lass of videosbelongingto the orpusprovidedbyJCT-VCandthePUsizes.
Table1:A tivationofWTMa ordingtovideo lassesand PUsizes HE LC 4x4 8x8 16x16 32x32 64x64 4x4 8x8 16x16 32x32 64x64 ClassA - X X X - - X X - -ClassB X X X X - X X X - -ClassC X X X - - X X - - -Class D X X X - - X X - - -ClassE X X X X - X X X - -ClassF X X X X - X X X -
-Table2:Relationbetweenintramodeandshapeoftemplate INTRAmode Shape
10 UDL
11 U
12 L
13 UL
2.2.2 Templateshapesignalling
The
4
template shapesare available. Consequently, two pie es of information must be signalledto thede oder: when ablo kis predi ted with WTMand whi h shape oftemplate wasused. Todoso,fourdire tion modeshavebeenoverloaded: from the mode10
up to themode13
. They areasso iatedto a shape ( f. Fig.1)aslisted in Table2. An extrabitis addedforallofthese four modesandset totrueifWTM is used asdes ribedinFig.4.Table3:Sear hareas hara teristi s
Blo kBsize Sear hwindowsnumber Sear hwindowswidth Sear hwindowsheight
4x4 3 12 4
8x8 2 20 8
16x16 2 8 16
32x32 2 4 32
2.2.3 Numberanddimensionsof sear h windows
Thenumberofsear hwindowsandalsotheirsize depond onthesizeof theblo kto bepredi ted.
The hara teristi softhesear hareasaresummarizedin Table3.
2.3 Conguration
TheWTMalgorithmiswrittenontopofthetestmodelofHEVC,release4.0. So,the onguration les dedi ated to WTM are basedon theHTM-4.0 all intra en oding ongurationle.
Ase tionisaddedtospe ifyparametersrelationtoWTM.Inparti ular,thisse tions indi ates whether:
•
WTMisa tivatedornot• 4
x4
,8
x8
,16
x16
,32
x32
,WTMpredi tionarea tivatedAnoptionalparameter,STMObserver, anbesettogeneratestatisti sorpredi tion maps.
Thefollowingex erptofa ongurationleliststheparametersrelatedtoSTM.
#============ WTM ================ STM : 1 # 0 : unsed, 1: a tivated
STM4x4 : 1 # Predi tion a tivated for 4x4 blo k size STM8x8 : 1 # Predi tion a tivated for 8x8 blo k size STM16x16 : 1 # Predi tion a tivated for 16x32 blo k size STM32x32 : 1 # Predi tion a tivated for 32x32 blo k size
STMObserver : 3 # 0:unused, 1: % of sele tion, 2: stats files, 3: output frames
Theotherse tionsofthe ongurationlearekeptun hanged.
2.4 Example of use
Tobuildthesofware,refertothe"how-to"providedinthedeliverable3.4anduploaded to thewebsiteofthePerseeproje t(http://persee.ir yn.e -nantes.fr/prive/).
TAppEn oder - tests. fg
where "tests. fg"isthe ongurationle.
Tode odetheen odedvideo,justenterthefollowing ommand:
TAppDe oder -b str.bin -o de .yuv
where "str.bin"isthe en oded video(the namewasspe iedin theen oding on-gurationle)and"de .yuv"thenameofthede odedvideo.
Referen es
[1℄ Persee,2d odingtoolsnalreport, ANR-09-BLAN-0170,DelivrableD 3.4,July 2013.