Proceedings Chapter
Reference
Data hiding capacity-security analysis for real images based on stochastic non-stationary geometrical models
VOLOSHYNOVSKYY, Svyatoslav, et al.
VOLOSHYNOVSKYY, Svyatoslav, et al . Data hiding capacity-security analysis for real images based on stochastic non-stationary geometrical models. In: SPIE Photonics West, Electronic Imaging 2003, Security and Watermarking of Multimedia Contents V . 2003.
Available at:
http://archive-ouverte.unige.ch/unige:47963
Disclaimer: layout of this document may differ from the published version.
1 / 1
stochastic non-stationary geometrical models
S.Voloshynovskiy, O.Koval,F. Deguillaumeand T. Pun
University of Geneva -CUI, 24 rue GeneralDufour, CH 1211,Geneva 4,Switzerland
ABSTRACT
Inthispaperweconsider theproblemofcapacityanalysis intheframework ofinformation-theoreticmodelofdata
hiding. Capacityis determinedbythestochasticmodelofthehostimage,bythedistortion constraintsandbythe
sideinformationaboutwatermarkingchannelstateavailableat theencoderand atthedecoder. Weemphasizethe
importance of propermodeling of image statistics and outline the possible decrease in the expected fundamental
capacitylimits, ifthereis amismatch betweenthestochasticimage model usedin thehider/attackeroptimization
game and theactual model used by theattacker. Toobtaina realisticestimation of possibleembedding rateswe
proposeanovelstochasticnon-stationaryimagemodelthatisbasedongeometricalpriors. Thismodeloutperforms
the previouslyanalyzed EQ and spikemodels in referenceapplication such asdenoising. Finally, wedemonstrate
howthe proposed model inuences the estimation of capacity for real images. We extend our model to dierent
transformdomainsthatinclude orthogonal,biorthogonalandovercompletedatarepresentations.
Keywords: watermarking,stochasticimage model, capacity,information theory,spikemodel,edgeprocessmodel.
1. INTRODUCTION
An important problem of digital data-hiding is the investigation of fundamental theoretical capacity limits, i.e.
somehowananalogtoShannon'slimitin digitalcommunications. TherecentworkproposedbyMoulinadvocatesa
game-theoreticapproachfortheevaluationofdata-hidingcapacity.
1
Inthescopeofthisapproach,thedata-hiding
capacityisconsideredtobeasolutionofamax-mingamebetweenthedatahiderattemptingatmaximizingchannel
capacityandtheattackeraimingatdecreasingthecapacity. Itis alsoassumednospecicform ofencoder/decoder
andattacking,butitisrathersupposedthatboththedatahiderandtheattackeraredoingthebesttoachievetheir
goals. Therefore, oneis looking for the maximum rate of reliablecommunications, overany possibledata-hiding
strategy,andanyattackthatsatisfythespeciedconstraints.
Anemergingpracticalproblem istheapplication ofthegame-theoreticparadignforthe analysisofdata-hiding
capacity of real images. The solution to this problem should provide the justication of many existing practical
algorithms and allow to fairly evaluate their performance and potential capabilities. An important aspect of this
problemistodevelopappropriatemodelsforattackingchannels,fordistortionmetrics,andforimagestatistics. The
analysisof thesethree itemshasgreatimpactonthesolutionofthemax-minproblem. Therefore,itisjustiedby
Moulinthataminimummeansquareerror(MMSE)estimatorofthehostsignalandaGaussiantestchannelfrom
rate distortion theory are the worst case attacks for the given constrained attack distortion D
2
. A squared-error
distortionmeasured(x;y)=(x y) 2
isselectedfortheanalysisdueto itswideusageincommunicationtheoryand
niceclosed-formresults. AGaussianmodelofthehostimageisselectedasasourcemodel,sinceitprovidestheupper
boundoncapacityfornon-Gaussiansourcesaswell. It isalsoassumedthat themaximumadmissible distortionfor
thedata hideris D
1
whilefor theattackeritis constrainedbyD
2
. This abstractscenarioofthe max-mingameis
shownin Figure1.
Animportantassumptionisthatboththedata-hiderandtheattackernotonlysharecompleteinformationabout
theirstrategies(besideasecretkeyusedbythedata-hiderforthewatermarkencryption/encoding/spatialallocation),
but also thestochasticmodelof thesource image. In theoriginal analysisof Moulin andMihcak, 2
it isproposed
touseanexpectation quantization(EQ) 3
andaspikemodel 4
toevaluatethedata-hidingcapacityofrealimages
due to their superior performance in the reference applications. Therefore, the nal watermark energy allocation
and theattack distortion distributionare performed accordingto theselected models. Although, this approach is
Hider D 1
Stochastic Image Model
Max-min Game
Model #1
Capacity
Attacker D 2
Stochastic Image Model
Model #2
Figure 1. Generalizeddiagramofgamebetweenthedatahiderandtheattacker.
intuitivelyjustied,it couldpotentiallyleadto overestimatetheactualcapacitysincetheobtainedestimateofthe
rateofreliablecommunicationsishighlysensitivetothesourcemodelselection. AlthoughtheEQandspikemodels
havealot ofadvantages,theydonotpreventthepossibilitythatnewmorepowerfulsourcemodelscanappear,and
thtthe obtainedfundamental capacity limitsshould bedetermined again. Therefore, theupper limitcharacter of
thegameapproachcan bequestionedin thiscase. Moreover,anevenworsesituation canhappeninpractice,when
theoptimalwatermarkenergy allocationis performedassumingtheEQor spikemodel (denotedasmodel number
1)inthescopeofthegameapproach,buttheactualattackerwillapplyanewmorepowerfulmodelforrealattacks
providingthesameattackdistortion D
2
. Thissituation isschematicallyshownin Figure2. Suchaformulationhas
alot incommonwithbroadcastingsystemsandcorrespondingproblemsofcapacityevaluation.
5
Encoder w y
x
b
Embedder
Channel
Decoder b ˆ D Worst Attack 1
D 2
D 1
Model 1 D Real Attack 1 D 2 Model 2
D Real Attack 1
D 2 Model 2
D Real Attack 1
D 2 Model M
Figure 2. Multichannelgamemodel.
Thegoalof this paperis to introduceanew class ofstochastic image modelsbasedon geometricalpriors that
havesuperiorperformancein somereferenceapplicationssuchasdenoisingasopposedtotheEQandspikemodels.
The proposed model is applied to thegame-theoreticset-up and newcapacitylimitestimates are obtained. Since
theproposedmodelhasconsiderablylowerlocalvariances,theobtainedpracticalembeddingcapacityisalsosmaller
than thoseobtained forthe EQ/spikemodels. Therefore, wedemonstratethat although the information-theoretic
game approach isan excellentframeworkfor providing theabsolute limitsof watermarkingcapacity, the practical
limits are highly sensitive to the model selection and to the transform domain. Essentially, we showthat actual
capacitymightbemuchlowerthanpredictedbytheEQorspikeimage models.
Section2briey reviews themain resultsof Moulin'sgame approachand newextended data-hiding gamesare
presentedinSection3. AnontrivialproblemofimagemodelselectionisdiscussedinSection4. Section5introduces
an edge process (EP) model and provides the comparison of this model with the EQ and spike models for the
issuesofgame-theoreticapproachandconcludesthepaper.
2. DATA-HIDING GAME-THEORETIC APPROACH
Weassumethatthemessagemisencodedbasedonasecretkeyintosomewatermarkw andembeddedintoahost
data(image)x. Theresultingstegodatay 0
isobtainedas:
y 0
=x+w : (1)
Weincludeinthisgeneralizedmodelbothspreadspectrumschemesandhostinterferencecancellationschemesbased
onpre-codingaccordingtoCostacodes. Wedenotextobeatwo-dimensionalsequencerepresentingtheluminance
oftheoriginalimage. Theith elementofxis denotedasx[i]where i=(n
1
;n
2
)and x2R N
and N =M
1
M
2 is
thesizeofthehostimage. Theadmissibledistortions areD
1
forwatermarkembeddingandD
2
fortheattacker.
2.1. Capacity for Gaussian channels
Theclosed-formsolution fordata-hidingcapacityhasbeenfoundfor i.i.d. Gaussian hostsignaland squared-error
distortionfunctions 6
:
C= ( 2
x
;D
1
;D
2 )=
1
2 log
2 1+
D
1
D
, ifD
1
<D
2
<
2
x
0;ifD
2
>
2
x ,
(2)
whereD = 2
x D
2 D
1
2
x D1
. Theoptimalattackisthecascadeofaminimummeansquareerror(MMSE)estimatorforx
giveny 0
,andofaGaussiantestchannelfromrate-distortiontheory. Inthecaseofsmalldistortions( 2
x
>>D
1
;D
2 ),
which is common in practical data-hiding applications and assuming large variance, we have D D
2 D
1 and
C= 1
2 log
2
1+ D1
D2 D1
,i.e. thecapacityexpressionisasymptotically independentof 2
x .
2.2. Capacity for parallel Gaussian channels: mainresults
We assume that the image is decomposed into K channels using some multirate transform. The grouped image
coeÆcientsin each channel are assumedto beindependent andall samplesin the channel havethesamevariance
andare i.i.d. N(0;
2
x
[k]). Letd
1
[k]and d
2
[k]bethedistortionsintroducedin channel kbythedata-hiderandthe
attacker,respectively. Thechannels areassumedtobeindependentandmemoryless. Theoptimaldata-hidingand
attackerstrategiesineachchannelarethoseforasingleGaussianchannelwithhostvariance 2
x
[k]andsquared-error
distortions d
1
[k]and d
2
[k]. The allocation of powersd
1
=fd
1
[k]gand d
2
=fd
2
[k]gbetween channels satisesthe
overaldistortionconstraints:
K 1
X
k =0 r
k d
1 [k]D
1
; K 1
X
k =0 r
k d
2 [k]D
2
; (3)
andtheinequaliatyconstraints
0d
1
[k]; (4)
d
1 [k]d
2
[k]; (5)
d
2 [k]
2
x
[k]; (6)
for0kK 1. Let ( 2
x [k];d
1 [k];d
2
[k])bethecapacityof channelkand P
K 1
k =0 r
k
=1.
Thewatermarkingcapacityforbothblindandprivateparallel-Gaussianwatermarkinggamessubjecttodistortion
constraints(D
1
;D
2
)isequalto 1
:
C=max
d
1 min
d
2 K 1
X
k =0 r
k (
2
x [k];d
1 [k];d
2
[k]) (7)
where
( 2
x [k];d
1 [k];d
2 [k])=
1
2 r
log
2
1+ d
1 [k]
d
2 [k] d
1 [k]
1 d
2 [k]
2
x [k]
(8)
wherer
= P
K
1
i=0 r
i
2[0;1]isthefractionofstrongsignalcomponentsandthemaximizationandminimizationare
subjectto theaboveoveralldistortion constraints(3-6). TheparallelGaussian watermarkchannelis schematically
k w
k W ¢ k
x
k k y
y¢
[ ] k k w [ ] k
x x
2 2
2
s s
s +
MMSE/MAP
k b ¢ 1
Figure 3. Moulin's optimal data-hiding and attack strategies for parallel Gaussin channels. Each coeÆcient
x[k] N(0;
2
x
[k]), 0 k K 1. Thedata hider allocates to each channel watermarkwith the power D
1 [k]=
f 2
w
[k]gresultingittotal powerD
1
. Itisassumed that w[k]N(0;
2
w
[k]).Theattackerintriducesto each channel
the distortion d
2
[k] resulting in total attacked image distortion D
2
. This distortion is distributed between the
MMSE/MAPestimatordistortionandthetestchanneldistortion. Theestimatordistortionisequaltothevariance
oftheestimator 2
e [k]=
2
x [k ]
2
w [k ]
2
x [k ]+
2
w [k ]
. Allchannelsaretreatedindependently.
2.3. Capacity for spike process model
Moulin and Mihcak 2
have used so-called spike process model introduced by Weidmann and Vetterli.
4
Under a
spikemodeltherearetwotypesofchannels: thosewithlargevariance,i.e. strongchannels 2
x
>>D
1
;D
2
,andthose
with low variance, i.e. weak channels 2
x
<< D
1
;D
2
. Theimage components are assumed to beverysparce and
independent. Assumethat 2
x
>>D
1
;D
2
,for0kK
1and 2
x
<<D
1
;D
2 forK
1kK 1.
Thecapacityforspikeimage modelisthendeterminedas 2
:
C= 1
2 r
log
2
1+ D
1
D
2 D
1
: (9)
Onecanmakesomeimportantconclusionsforthecapacitybasedonthespikemodel.
(1)Thecapacitydoesnotdependon 2
x
[k]forstrongchannels.
(2)AlmostallenergyofthewatermarkD
1
isallocatedtothestrongchannelsandtheper-samplecapacityisthe
sameforallstrongchannels.
(3)The attackercannot considerablyattack thestrong channels. Theextreme caseof theattack isto put the
strongchannelsto zero(or todecreasetheirvariance). However,this leadstostrongdegradations(likeintherate-
distortiontheory). Intheextremecaseofzero-rateallocation,itwillbeequalto thevarianceofthestrongchannel
2
x [k].
TheMMSEltercannotconsidereblyhelpkillthewatermarkinthestrongchannels,becauseitdoesnotproduce
anaccurateestimateofthehostsignal. ThevarianceoftheMMSE/MAPestimatorsintheGaussiancaseforboththe
hostimageandthewatermarkis determinedas 2
MMSE [k]=
2
x [k ]d1[k ]
2
x [k ]+d1[k ]
. Accordingto thespikemodelassumption
for the strongchannels 2
x
[k] d
1
[k]. Therefore, 2
MMSE
[k] ! 1and the MMSE estimatorterm
2
x [k ]
2
x [k ]+d
1 [k ]
!1
resultingin x[k]^ !y 0
[k] asone-to-onemapping,i.e. noreliableestimate isproduced in thiscase. Theonlyattack
thatisintroducedin thiscaseis atestchannelfrom therate-distortiontheory.
3. EXTENDED DATA-HIDING GAMES
Theoriginal information-theoreticdata-hidinggameproposedby Moulinand Mihcakassumesthat boththedata-
hider andtheattackershare thesamestochastic image modelfortheir max-minstrategies. Moreover,aparticular
form of stochastic image model, i.e. EQ or spike model, is assumed to tackle the sparcity of transform image
coeÆcients. Therefore,theobtainedresultsrefertooneofthesemodels. Wepresentheremoregeneralset-upthat
canbevalidformanypracticalapplications.
Firstof all, anatural questionconcerns the capacityestimate obtained basedon the EQ/spike model. Is this
Thirdly,wetackletheproblemoftheobtainedcapacitylimitrobustnesswithrespecttothevariationsofstochastic
image models. We will follow further the above parallel Gaussian source set-up. The particular interest is to
investigatethevariabilityoftheobtainedcapacityestimatesforthedierentimagemodels. Letusrstassumewe
canndthedata-hidingcapacityC IM
1
forimage modelIM
1
(forexamplethespikemodel):
C IM
1
= max
fd1;IM1g min
fd2;IM1g K 1
X
k =0 r
k (
2
x [k];d
1 [k];d
2
[k]); (10)
and for someother model IM
2
that is feasible forthe given image and provides thememorylessand independent
channeldecomposition:
C IM
2
= max
fd1;IM2g min
fd2;IM2g K 1
X
k =0 r
k (
0 2
x [k];d
1 [k];d
2
[k]); (11)
where 0
2
x
[k]isthelocal varianceforthemodelIM
2
assumedtobealsoi.i.d. locally Gaussian.
Thesecond possiblescenario we encompassis basedon taking into account three strategies, bydistinguishing
those of thedata-hider at theencoder, theattackerand thedata-hiderat thedecoder. Weassume that aproper
channel stateestimation(CSE) isperformedthat isableto completely characterizetheattackchannel in termsof
geometricalsynchronization,fadingandnoisedistribution.
7
Therefore,thedata-hiderat thedecoderisabletouse
this information to maximize the channel capacity. In fact, this scenario is commonly used in practice in digital
communicationsandhasthesamepracticalimportancefordata-hidingtechnologies.
Imagineapracticalsituationwheresomewatermarkingtechnologyisdevelopedandmaintainedduringacertain
time on the market. Thedesign of this technology assumed that the data-hiderand the attacker share the same
imagemodelasthesolutionto:
C IM
1
CSE
= max
fDH
encoder
;d1;IM1g
min
fAtt;d2;IM1g
max
fDH
decoder
;IM1g K 1
X
k =0 r
k (
2
x [k];d
1 [k];d
2
[k]); (12)
whereDH
encoder
;AttandDH
decoder
denotestheadmissiblestrategiesofthedata-hiderattheencoder,theattacker
and thedata-hiderat thedecoder,respectively. A certainamountof imagesis watermarkedusing thistechnology
and from themarketing perspectivethe watermarkdecoder, designed assuming the image model IM
1
, should be
maintainedonthemarketduringsometime.
However,amorerealisticscenarioistoassumethattheattackerisfollowingtherecenttrendsinimageprocessing
and possess some more powerful image model IM
2
that outperforms the model IM
1
at least for some reference
applications such as image denoising or compression. We formulate here two possible sub-problems, considering
eitheranon-informedoraninformeddecoderwithrespecttotheimagemodelIM
2
used bytheattacker.
Therst sub-problem assumesa non-informed decoder, i.e. no side information about the model used by the
attackerisavailableforthedecoder:
C
non informed
CSE
= max
fDH
encoder
;d
1
;IM
1 g
min
fAtt;d
2
;IM
2 g
max
fDH
decoder
;IM
1 g
K 1
X
k =0 r
k (
2
x [k];
0 2
x [k];d
1 [k];d
2
[k]): (13)
Therefore,thejointpairencoder-decoderdoesnotevenknowwhichmodelwasusedbytheattacker,butthedecoder
still assumes the rst model. It is an interesting problem to estimate the loss in performance with respect to
thismismatchdetermined byrelativeentropyD(p
IM1 kp
IM2 ))=E
pIM
1 log
p
IM
1
pIM
2
betweentwop.d.f.s p
IM1 andp
IM2
correspondingtoIM
1
andIM
2
, respectively.
Finally, regarding the second sub-problem, we can assume that the technology provider is also following the
recenttrendsin stochasticimagemodelingandcanintegrateanewdecoderthat assumesbothpossiblemodelsfor
theattacker. However,anessentialamountofimagesisalreadywatermarkedforthemodelIM
1
andthetechnology
providershouldgaranteethemaintenanceofthecopyrightorwatermarkdetectioningeneral. Inthiscase,thevalid
formulationis:
C
informed
CSE
= max
fDH
encoder
;d
1
;IM
1 g
min
fAtt;d
2
;IM
2 g
max
fDH
decoder
;IM
1
;IM
2 g
K 1
X
r
k (
2
x [k];
0 2
x [k];d
1 [k];d
2
[k]): (14)
(11).
4. IMAGE MODEL SELECTION
The model selection is a very important but at the same time a very ambiguous problem. It involves a lot of
subjectiveexperiencedealingwiththeestimation,detectionandrate-distortionproblems. Therefore,thisprocedure
cannotbeobjectiveorautomatic.
This conditions the necessity to justify the fundamental capacity limits for the EQ and spike models. The
selectionoftheEQorspikemodelsis explainedbytheirexcellentperformancein somereferenceapplicationssuch
asimage compression and denoisingand good t to theparallel Gaussian channel model. In amoregeneralcase,
the advantages of onemodel overanothermodelare considered basedon thesatisfaction ofa listof requirements
whichdeterminethesuitabilityofthemodelforthepracticalapplications:
(a)modelsimplicity(preferablyGaussian-typemodelsduetoeasyintegrationanddierentiation);
(b)modelabilitytoleadtoaclosed-formanalyticalsolution;
(c)model andresulttractabilityandexistanceofperformancebounds(preferablyGaussian-typemodelsdue to
theupperandlowerbounds inchannel capacityand rate-distortiontheory);
(d)modelrobustnessandapplicabilitytoawideclassofrealimages.
5. EDGE PROCESS (EP) MODEL
5.1. Model denition
Weconsider animagex withasupport S asarealizationofarandomeld X withdistinct stochasticbehaviorin
dierent regions. Let R
1
;R
2
;:::;R
M
bea partition of the support S of x, i.e., R
i
\R
j
=;;i6= j and [
i R
i
=S.
Let x
l
denotes thesubsetof image pixels supported bytheregions R
l
. Inourmodel, weassume that each region
R
l
is fully coveredbythemodel
l 2f
1
;
2
;:::;
L
gand that notwoneighboring R
l
containthe samemodel. In
particular,weassumethatthepixelsintheimagesubregionx
l
aredistributedwithjointprobabilitydensityfunction
(pdf)p
x (x
l j
l ).
Ourgoalis tointroducetheedgeprocessmodelandto compareitwith theEQmodel. TheEQmodelbelongs
to the class of intraband stochastic image models and assumes that the wavelet coeÆcients are Gaussian (in the
originalpaperofLoprestoaGeneralizedGaussian 3
)distributed,withzeromeanand variancesthat dependonthe
coeÆcientlocationwithineachsubband. It isalsoassumedthatthevarianceisslowlyvarying.
WeassumethateachsubbandofmultiresolutioncriticallysampledtransformhasitsownsupportS
l
,l=1;:::;3M,
whereM isthenumberofdiadicdecompositionlevelssuchthatS
i
\S
j
=;;i6=jand[
l S
l
=S. Fornon-decimated
wavelet transformwithout downsampling used in our modeling, eachsubband hasthe samesupport as above but
thedimensionalityofeach S
l
isthesameastheoriginal image. Accordingto thepartitionapproachapplied tothe
EQ model, weassume that onlyone regionR
l
is given within the subbandS
l
and allcoeÆcients in this subband
belongtothesameregionR
l :
R
l
=fx
l :x
l
[i]N(0;
xl 2
[i])g (15)
and allcoeÆcientsareconsidered tobeindependentidentically distributed (i.i.d.) with Gaussian distribution,but
withdierentlocalvariances
xl 2
[i]. Equivalently,thismeansthatonlyonestochasticmodeloutoff
j
gisapplied
tothewhole supportS
l andp
xl (x
l j
l
)followsani.i.d. Gaussianp.d.f.. Inthefollowing,wewillconsideronlythe
imagemodelforonesubband.
ContrarilytotheEQmodel,theedgeprocessmodelassumestwodistinctivesetsofcoeÆcientsinwaveletdomain
for each subband, i.e. those belonging to the at regions and those belonging to the edge and texture regions.
Moreover,it is assumedthat atransition correspondingto an edge orto afragmentof texture consists of several
distinctmeanvaluesthatpropagatealongthetransition. Inthefollowingwewillrefertothetransitionsimplyasthe
edge. ThesemeanvaluescouldbeconsideredasthereconstructionlevelsoftheoptimalLloyd-Maxscalarquantizer.
ThevariationofthecoeÆcientswiththesamemeanissupposed tobelowalongtheedge. Accordingto theabove
partitionapproach,theedgeprocessmodelisdened as:
R
1
=fx:x[i]N(0;
2
[i])g;R
2
=fx:x
j
[i]N(x
j [i];
x 2
[i])g; (16)
1 2 1
assumed to bezero-mean Gaussian random variables with thelocal variance 2
x
[i]. The region R
2
corresponds to
thetextureandedgeregions. Eachdistinctivegeometricalstructurecorrespondingtotheedgeortexturetransition
within R
2
is decomposedin asetof localmeanconstellations. Moreover,aparticularmean valuex
j
[i],j =1;:::;J
propagatesalongtheedgecreatingtheso-callededgeprocess(EP).Therefore,coeÆcientsontheedgeareconsidered
tohaveoneofthepossiblemeanvaluesfromthesetfx
j
[i]g,contrarilytotheEQmodelwhichdoesnotdierentiate
atand edge regions and assumeszero-meanfor allcoeÆcients. Moreover,wecanalso assume that thevariation
of the coeÆcients with respect to the mean values(and this is especially true for the overcompletetransform) is
verysmall. Therefore,theEPmodel assumesthat theimage consistsof randomGaussian processwith zero-mean
and somesmall local variancefor theatregionswithalmost "deterministic"edge occlusionswhich haveaclearly
denedgeometricalstructuredependingonthemutualorientationoftheedgeandthesubband. Moreover,normally
transitions alongthe edgehave longerstationary lengththan thetransitions within thetexture (that explainsthe
existenceofhighercorrelationsalong theedges);thisprovideshigherredundancyof thesupportfor moreaccurate
model parameter estimation. Due to this fact, thestationarity conditionis more strict forthe edges than forthe
textures. Finally, allthis leads to theconclusionthat thereal varianceof the subbandsis verylow andis mostly
determinedbytheatregions,contrarilyto theEQmodelorevenmoreforthespikemodelwherethehuge spikes
ofimagecoeÆcientswithlargevariancecanoccurduetotheedgethatissupposedtomodelthewaveletcoeÆcients'
sparcity. Norelationshipor special geometricalspatial structure is assumed among thespikes contraryto the EP
model wherethe"spikes"belongingto thesameedgearetreatedjointlyalongthedirectionofedgepropagation.
5.2. Model parameters' estimation
Thegoalofthissectionistoanalyzetheproblemofmodelparameterestimationandtopointoutthemaindrawbacks
oftheEQ/spikemodels. Weassumethatamaximumlikelihood(ML)estimationisusedtoestimatethelocalvariance
fortheEQmodelinsomelocal neighborhood(k). WeassumeaLxLsquarewindow(k)centeredat locationk.
TheestimationofthelocalvarianceaccordingtotheMLestimateis:
^ 2
x [i]=
1
jj X
2(k ) jx[]j
2
(17)
where j j is thecardinality of . Although this ML variance estimatoris widely used in the image processing
community,itsusageisjustiedonlyforstationarydatawhichisnotthecaseforthewaveletcoeÆcients. Moreover,
the ML variance estimator is only asymptotically unbiased, i.e. E[^
2
x ] =
2
x 1
jj
2
x
. The bias in the variance
estimation is 1
jj
2
x
and to reduce the bias oneshould increase the sampling space , i.e. lim
jj!1 E(^
2
x ) =
2
x .
However, this contradicts to the non-stationary nature of wavelet coeÆcients. The condition of stationarity is
especially violated in thevicinity ofedges and textures. As aresult, alot of outliers from thewaveletcoeÆcients
withdierentlocal meansarein thesquarewindowandoneobtainsextremelyhighestimatesoflocalvariance. To
avoidthis negativeeect,oneneedsto reducethewindowsize,butthiscanleadto anincreaseofthebias. Thatis
whyan adaptivewindowselection basedonabootstrapmethod wasused to achieveacompromisebetweenthese
twoconictingrequirements.
8
Therefore,thelargewindowwas selectedforthe stationarycoeÆcientsin atregions andthe windowsize was
decreased for the edge and texture regions. However, even in this case, the smallestlocal window of size of 3x3
cannotgaranteeareliablevarianceestimatefortheedgeregionsduetocriticalbiasandnon-stationarycoeÆcients
inclusionespeciallyforthedetailsubbandswhichwillcontainlargepositiveandnegativesampleseveninthesmallest
possiblewindowsize. ThissituationisschematicallyexplainedinFigure4. Figure4arepresentssomeedgestructure
in the transform domain without downsampling and Figure 4b the same edge structure but after downsampling.
The ML estimate corresponding to equation (17) is shown in Figure 4c for the edge wavelet coeÆcient. One can
clearlyobservetheabovemain problemoftheMLestimatedueto thenon-stationarityofwaveletcoeÆcients. The
local windowcontains both thecoeÆceintsbelonging to theedge and thecoeÆcients belonging to the atregion.
Contrarily, theEPmodel(Figure 4d) computesthemean onlyalong thestationary edge that correspondsto the
stationarityconditionfortheML-estimate,i.e. ittakesonlythosecoeÆcientsbelongingtothesetR
2
foraparticular
meanconstellation.
S
(a)
S
(b)
set for estimation
S
(c)
S
set for estimation
R 1
R 2
(d)
Figure 4. Explanation of theEQ and edge process EP models: (a) the edge structure in thetransform domain
(overcompleterepresentationwithoutdownsampling);(b)downsampledsparsedatainthecaseofcriticallysampled
transform (wavelets); (c) ML-estimation of the local image variance for theedge coeÆcientin the local set using
a globally stationary assumption; (d) ML-estimation of the local image variance for the edge coeÆcient using a
stationaryset ofcoeÆcientsonlyalongtheedge.
5.3. EP model verication on real images
TodemonstratethemainfeaturesoftheEPmodelonrealimageswehavechosentheimageLenaofsize512x512and
decomposeditusinga5-levelwavelettransformwithDb8lter(Figure 5a). TheEPmodelaimsatamoreaccurate
modeling of edges in the transform domain. As an example, we have selected a fragment of the rst horizontal
subbandontheshoulderofLenaimage(Figure5b)thatcorrespondstoawelldistinguishedpropagatingedge. One
canalsoobservesomevariationsalongtheedgedue tothecritically samplingcharacterofwavelet transform. The
edgestructurewillbesmoother,i.e. withsmallervariance,fortheovercompletetransformwithoutdownsampling.
9
SmallvariationsofwaveletcoeÆcientsareobservedin theatregionsonbothsidesoftheedge. Theestimationof
themeanalongtheedgeaccordingtotheEPmodel,followedbyitssubtraction,resultsinthemoreuniformrandom
eldshowninFigure5c. Itis importanttonote thattheamplitudeofthecoeÆcientsisconsiderablyreduced. The
"edge"isnotanymorevisuallydetectable andthefragmentlookstobemoredecorrelated.
Importantchanges are alsoobserved for the subband histograms(Figure 6). An essentially non-Gaussianhis-
togramof the originalsubband (Figure6a) istransformed into ahistogram withperfect Gaussian t (Figure6b).
Therefore,besidesthefactofadditionaldata"decorrelation",theEPmodelalsoproducestransformcoeÆcientswith
Gaussianp.d.f.. SinceboththeEQandspikemodelsrefertothelocalvarianceasthemodelparameter,wevisualize
thesubbandlocalvariancesinFigure7afortheEQmodelandinFigure7bfortheEPmodel. 8-meanconstellation
wasusedintheEPmodelfortheR
2
region. Obviously,thissimpleconstellationschemecausessomeapproximation
errorthatgivesrisetoanincreaseinlocalvarianceinthevicinityoftheedge. Morepowerfulshapeapproaximation
techniques can be used for this purpose and the work is underway. The ML variance estimate was used in both
cases. One canalso observe asignicant decrease of the local image variance, roughly 150times with respect to
the variancepeak values. The decrease of thevariance hasa crucial impact onthe performance ofdenoising and
compressionalgorithmsaswellasonthecapacityestimationproblem.
Todemonstratethemodelabilitytocarryoutandto captureonlysignicantimage componentswithacertain
levelofsparcity,weperformedaset ofexperimentsshowninFigure8. First,theimagewasdecomposedusingDb8
waveletsin5-levelpyramid. Second,theEPmodelwasappliedandtheedgeswerereplacedbytheirmeanestimates
accordingtotheEPmodel. Figure8ashowstheimagereconstructedbasedonthecompletelypreservedinformation
aboutatregionsandthemeanestimatesalongtheedges. Figure8bisreconstructedonlybasedontheestimateof
meansandzerosinallremainingpositions(totalnumberofnon-zerocoeÆcients66660outof262144forthe512x512
original image). The obtainedresults demonstrate thehigh sparsityof the EP model and also the delity to the
originaldata. Itdemontratesthepossibility todesignnewmorepowerfulcodersbasedonshapecoding tocapture
theedgestructures.
(a)
0 10
20 30
40 50
0 10 20 30 40 50
−60
−40
−20 0 20 40 60 80
(b)
0 10
20 30
40 50
0 10 20 30 40 50
−8
−6
−4
−2 0 2 4 6 8
(c)
Figure5. ExplanationoftheEPmodel: (a)5-levelorthogonalwavelettransform(Db8)appliedtoimageLena;(b)
asubbandfragmentcorrespondingtotheedgeontheLena'sshouldertakenin thersthorizontal subbandand(c)
thesamefragmentafter EPmodelmeansubtractionalongtheedge.
−40 0 −30 −20 −10 0 10 20 30 40
0.05 0.1 0.15 0.2 0.25 0.3 0.35
(a)
−10 0 −8 −6 −4 −2 0 2 4 6 8 10
0.005 0.01 0.015 0.02 0.025 0.03 0.035 0.04 0.045
(b)
Figure6. Histogramsoftherstlevelhorizontalsubdandbefore(a)andafterEPmodelmeansubtraction(b).
Therefore,wecan briey summarize themain features of theEP image model. First,the EPmodel oersan
additionaldata"decorrelation"evenforthexedtransformbasisfunctions,whichcouldbeaveryusefulfeaturefor
manyapplications such as denoising,compression and watermarking. It should alsobenoted that thecomplexity
of this "decorrelation" transform, besides model overhead,still remains almost the sameas thecomplexityof the
corresponding wavelet or overcompletetransforms. Secondly, theresultingdistribution ofthe subbandcoeÆcients
is Gaussian asitis demonstratedin Figure6. Thisfurther considerably simplies theanalysis and guarantiesthe
existenceofclosed-formsolutionsformanyapplications, contrarillytonon-Gaussianimagemodels. Sincethedata
isGaussian anddecorrelatedthisalsobringsanadditionalbenetfortheindependentcharacterofthecoeÆcients.
Thisallowsmodelingjointsubbandp.d.f. asaproductofindependentp.d.f.sofeachcoeÆcient. Theideatousethe
parallelGaussianchannelfortheanalysisofdata-hidingcapacity. Thirdly,thesubractionofthelocalmeanalongthe
edges makesdata"stationary"andconsiderably reduces thevarianceestimation basedontheML-estimator. This
hasanimportantimpactontheperformanceofdenoisingandcompressionalgorithmsaswellasprovidesacompletely
dierentjusticationfordata-hidingalgorithmperformanceopposedtotheEQmodel(smallerhostinterferencefor
spreadspectrumbasedwatermarkingtechniquesanddierentcapacityestimationlimits). Inaddition,theEPmodel
providesnewpossibilitiesforthedesignof advanced watermarkingattacks.
2000 4000 6000 8000 10000 12000
(a)
20 40 60 80
(b)
Figure 7. Varianceestimationin wavelet domainforDb8forthesecond horizontal subband(increasedinsize for
visualization): (a)MLvarianceestimationina5x5windowusedintheEQ/spikeprocessmodeland(b)MLvariance
estimation for the EP model computed in a5x5 window. Note that the ranges of variancevalues decrease from
[0..13000](EQ/spikemodel)downto[0..80](EPmodel).
(a) (b)
Figure 8. Reconstrunction of the Lena image in wavelet domain for Db8: (a) from the meansof theEP model
andcompletelypreservedinformationaboutatregions(PSNR=38.90dB)and(b)onlyfromthemeansoftheEP
model (PSNR=36.95dB).
PSNR[dB].
Imagemodel Standard deviation of noise Wavelet type
10 15 20 25
L E NA
Noisyimage 28:13 24:63 22:10 20:17
EQ(xedwindow) 34:44 32:21 30:50 29:18 Db8
Mihcak(bootstrap) 34:58 32:59 31:17 30:06 Db8
EP(positions) 34:73 32:78 31:36 30:23 Db8
EP(positions) 36:21 34:44 33:00 31:83 9=7;overcomplete
EP 36:52 34:81 33:55 32:53 9=7;overcomplete
B A R B AR A
Noisyimage 28:14 24:63 22:11 20:18
EQ(xedwindow) 32:97 30:45 28:73 27:38 Db8
Mihcak(bootstrap) 32:84 30:46 28:86 27:65 Db8
EP(positions) 33:09 30:87 29:31 28:16 Db8
EP(positions) 34:83 32:83 31:41 30:23 9=7;overcomplete
EP 35:05 33:16 31:92 30:78 9=7;overcomplete
5.4. EP model performance in a referenceapplication: denoising
An importantaspect ofthe model selectionisits performance in somereferenceapplications. Wehavechosenthe
problem of removal of the AWGN from images as a reference application due to its simplicity and existence of
benchmarkingresults. Since, theEQmodelis used in thedenoising methodof Mihcak et al 8
wehave chosenthis
methodforthesakeofcomparision.Moreover,toavoidanysubjectiveimplementationissues(extractionofedges)of
aparticular modelparameterestimationwehaveusedanempirical upperboundasacriterionforthecomparison.
Theempiricalupperboundassumesthatthestochasticmodelparameterestimationisperformedbasedontheclean
originalimage. InthecaseoftheEQmodelitreferstotheestimationofthelocalimagevariancesforeachsubband.
Moreover,wehaveinvestigatedtwomodicationsoftheEQmodels,either usingaxedwindowforallcoeÆcients,
or using a bootstrap versionof the EQ denoiser described by Mihcak et al.
10
In the case of the EP model, we
assumethat theinformationaboutthe edgepositionsis available andthelocal varianceisdirectly estimated from
thenoisyimage intherstsetofexperiments. Thesecondexperimentassumesthatthelocalvarianceisestimated
fromtheoriginalimageforfairbenchmarkingwithEQmodel. Moreover,to demonstratetheupperlimitoftheEP
model performance, wealsoperformedthedenoisingin theovercompletedomain. Theempiricalupperboundsare
showninTable2fortwotestimagesLenaandBarbara. MoreresultsconcerningtheEPmodelperformancecanbe
foundinourpaper.
9
Therefore,itisobviousthattheEPmodeloutperformstheEQmodelindenoisingapplications
byabout0.3-1dBforthecriticallysampledwaveletsandthegapisincreasedupto2.8-3.4dBfortheovercomplete
domain. Thismeans that notonlymodel selectionis important,but also that thetransformdomain could havea
signicantimpactonthecapacityestimationproblem. Inconclusion,theusageoftheEPmodelasopposed tothe
EQmodelisjustiedaccordingto thisreferenceapplication.
6. CAPACITY FOR THE EP MODEL
AssumingtheparallelGaussianchannelenergyallocationforthespikemodel,weestimatethedata-hidingcapacity
inthecaseoftheEPmodel. Weusethetechniqueproposed byMoulinandMihcak 2
forthispurpose. Thecapacity
isestimatedfortheattackerdistortionD
2
=2D
1
andforD
2
=5D
1
forseveraltestgrayscaleimagesLena,Barbara
and Baboonof size 512x512(Table2). The resultsfor thespikemodel are notsurprising. The images with large
amountoftransitionslikeBaboonandBarbaraarecharacterizedbylargef 2
x
[k]gandalargeembeddingdistortion
D
1
forBaboonimage. ThusthecapacityestimateishigherthanfortherathersmoothimageLena. TheEPmodel
producesmuchlowerestimatesofdata-hidingcapacityduetoamoreaccurateassumptionsaboutimagestatisticsin
distortionD
1
forthespikeandEPmodels.
Image D
1
D2=2D
1
D2=5D
1
NC(spike) NC(EP) C(spike) C(EP)
Lena 10 31855 7857 6951 1026
Barbara 10 54632 8480 13045 1237
Baboon 25 80952 2568 17562 436
thetransitionregions. SincethevarianceoftheEPmodelissignicantlylowerintheseregions,whicharesupposed
tobethemaincarriersofthewatermarkaccordingtothemax-minapproachandEQ/spikemodel,thedierencein
thedata-hidingcapacitybetweenthespikeandtheEPmodel isconsiderable. Onecanarguethat thecapacityfor
theEQ/spikemodel ishigherandthus itshould beconsideredasanupperbound. ThedierencebetweentheEQ
modelcapacitiesinwaveletandDCTdomainswasexplainedbythefactthatwaveletsproducebetterindependence
betweenthecahnnels andbettert to Gaussiandistribution.
2
However,comparing theEPand EQmodels,asit
was shownisSection 5,onecanclearlyobservethat theseconditionsareevenbettermetforthe EPmodelrather
that fortheEQ model. Therefore, theexistenceoflargercapacities fortheEQ modeldoesnotimmediatelymean
thatthese resultsshould beconsideredtobetiedtopracticallyreachableupperbounds.
7. ATTACK BASED ON EP MODEL
ThestatisticsofrealimagesundertheEPmodelcanalsobeusedtodevelopamoreinvolvedattackingstrategy. The
main ideaofthisattackis basedontheniceapproximatingfeatureof theEPmodel,asdemonstratedin Figure8.
Wehaveshownthat byonlypreservinginformation aboutmeansalongtheedges, onecanobtainanimage ofhigh
quality. Therefore,insteadof"killing"therateofspikecoeÆcientswithlargevarianceaccordingtotheGaussiantest
channel andassumption aboutzero-mean,weproposeto setto zeroallatregionsand to replacealledgesbythe
correspondingmeanestimates,thuscompletelykillingthewatermarkintheseregions. Thisattackisanasymptotic
caseoftheGaussiantestchannelforzero-rate. Inthiscase,theattackdistortionwillbeproportionaltothevariance
oftheedgeprocess,thatisrelativelysmallinthecaseoftheEPmodelcontrarytothezero-meanEQmodel.
IfweassumesomeedgewaveletcoeÆcienttobeaconstantvaluealongtheEPdenededgestructure,andthat
amax-minenergyallocationforthei.i.d. watermarkwwithenergy 2
w
isperformed,weobtainthemodel:
y[i]=x[i]+w[i]=+w[i]: (18)
Assumingthewatermarktobei.i.d. Gaussianw[i]N(0;
2
w
),wecanapplytheMLestimatefor:
^
ML
= 1
j 0
j X
j2 0
y[j]; (19)
where 0
isasupportoftheedgeandj 0
jisthelengthoftheedge. TheMLmeanestimateistheunbiasedestimate
E[
^
ML
]= withthe varianceof theestimateVar (
^
ML
)=Var (j
^
ML j
2
)= 1
j 0
j
2
w
. Therefore, thelongerthe
edge j 0
j, themoreaccurate meanestimate onecanreceive. In thecaseof an i.i.d. Gaussian assumptionabout
regionR
2
(see equation(16)) x[i]N(;
2
x
), thevarianceof themeanestimator of will be Var [
^
ML ]=
2
x +
2
w
j 0
j .
Inthe caseof asimplereplacement ofi.i.d. x[i]N(;
2
x
)by thesample meanestimated from the watermarked
image, theresultingvarianceofthe replacementerror willconsist ofE[j
^
ML xj
2
]= 2
x +
2
w +
2
x
j 0
j
that represents
theabovementionedasymptotic caseofzero-rate. Although thisattackcompletely kills thewatermarkin theat
regionsandreplacestheedgestructuresbytheEPmodelmean,theinroduceddistortionwillbesmallerthanthose
forthe EQmodel andGaussian testchannel. Thisis dueto thefact that sinceoneexploits theredundancyalong
thestationaryedgetogetmoreaccurateestimateofthemeanandlowerlocalvariancealongtheedge(contrarilyto
theEQ/spikemodelswhereallcoeÆcientsaretreatedseparately).
Wehaveconsideredtheproblemofdata-hidingcapacityforrealimagesbyapplyingresultsobtainedbyMoulinand
Mihcakforparallel Gaussianchannels. A newstochastic imagemodel hasbeenintroduced forthewaveletdomain,
the so-called EP-edge process model, that moreaccurately treats data in the regions of edges and textures. We
emphasizethecrucialroleofmodelselectionondeterminingcapacity. Although,theintroducededgeprocessmodel
provides an even moreaccurate image modeling, that has beenproven in a reference application, and ts better
the conditions of proper image decomposition in the parallel Gaussian model, the obtained results considerably
deviatefrom those obtainedfortheEQ/spikemodels arguedto be"upperbounds" onactualcapacity. Therefore,
the objectivecharacterof these bounds fordetermining thecapacityof realimages is under question. We havein
fact shownthat capacitylikelyto bemuchlowerthan previouslythought. Wealsodemonstratetheimportantrole
of model mismatch in the extendeddata-hidinggames. Finally, anew attack basedon theproposed EQ model is
presentedinthepaper.
ACKNOWLEDGMENTS
ThispaperwaspartiallysupportedbytheSwissSNFgrantNo21-064837.01"Generalizedstochasticimagemodeling
for image denoising, restoration and compression", and by the CTI-CRYMEDA-SA and Interactive Multimodal
Information Management (IM2) projects. Theauthors are thankfulto KivancMihcak (Microsoft Research, USA)
forhelpfulandinterestingdiscussions.
REFERENCES
1. P.Moulin,\Theroleofinformationtheoryinwatermarkinganditsapplicationtoimagewatermarking,"Signal
Processing, Special IssueonInformationTheoretic IssuesinDigitalWatermarking81(6),pp.1121{1139,2001.
2. P.Moulin and M.K. Mihcak, \Aframeworkfor evaluating the data-hidingcapacityof image sources,"IEEE
Trans.onImage Processing11,pp.1029{1042,September2002.
3. S.LoPresto,K.Ramchandran,andM.Orhard,\ImagecodingbasedonmixturemodelingofwaveletcoeÆcients
andafast estimation-quantizationframework," in DataCompressionConference 97, pp.221{230,(Snowbird,
Utah,USA),1997.
4. C.Weidmann andM.Vetterli, \Rate-distortionanalysis of spikeprocesses," in DataCompression Conference,
(Snowbird,USA),March1999.
5. S. Voloshynovskiy, F. Deguillaume, O. Koval, and T. Pun, \Robust digital watermarkingwith channel state
estimation,"SignalProcessing,2003. submitted.
6. P.MoulinandJ.O'Sullivan,\Information-theoreticanalysisofinformationhiding,"PreprintsubmittedtoIEEE
Trans.onInformation Theory, October1999. availablefrom http://www.ifp.uiuc.edu/moulin/paper.html.
7. S. Voloshynovskiy, F. Deguillaume, S. Pereira, and T. Pun, \Optimal diversity watermarking with channel
stateestimation,"in IS&T/SPIE's Annual Symposium, Electronic Imaging 2001: Security andWatermarking
ofMultimedia ContentIII,vol.4134,pp.23{27,(SanJose,CaliforniaUSA), 21-26January2001.
8. M. K. Mihak, I. Kozintsev, and K. Ramchandran, \Spatially adaptive statistical modeling of wavelet image
coeÆcientsanditsapplicationtodenoising,"inIEEEInternationalConferenceonAcoustics,SpeechandSignal
Processing, (Phoenix,USA),March1999.
9. S. Voloshynovskiy, O. Koval, and T. Pun, \Wavelet-based image denoising using non-stationary stochastic
geometrical image priors," in IS&T/SPIE's Annual Symposium, Electronic Imaging 2003: Image and Video
CommunicationsandProcessingV, (SanClara,CaliforniaUSA),20-24January2003.
10. M. K. Mihcak, I. Kozintsev, K. Ramchandran, and P. Moulin, \Low-complexity image denoising based on
statisticalmodelingofwaveletcoeÆcients,"IEEESignalProcessingLetters6,pp.300{303,December1999.
