the Problem

Some Contributions to the Change Point Problem

©ChithranVadavcrkkoLVasude\'atl

Abstract

model (Hawkinsetal. (2003)) nnd the modified information criterion (Chenetal (2006» rely on tbeparametricdistributionofthequalitychara.cteristic, andallYde-

empirical~likelihood-basedinformation criterion (ELIC) for idcntifyillgchanges in the

Acknowledgements

I want to sincerely acknowledge the financial support provided by the School

Iamgrateful to Dr. P.G.Sankaran, Dr. N. Balakrishna(Cochin University of ScienceandTecbnology,lndia),andProLP.K.Venugopalan(SrceKeralaVnrmaCoI- lege, Thrissur1India) for referring me to the Memorial University of Newfoundland

AsokanMulayath Variyath.andDr. J.G. Loredo-Ostirorintroduciugmetovarious

Table of Contents

3.1EmpirialLil«lihood 1LJPt<>filoEmpiri<lolUb!ibood

3.3EL-Based Information Criterion .

3.5Computational Algorithm

4.3Simula.tion Studies for Normally Distributed Data (Univariate)

4.3.3 Cuse3:BinomiaIPriorforq(s)

8 Coodudl,..IUm...1oo

List of Tables

1IC\;bao1\-_lorloUC_ELIC'

( I 01._01 . . . . , - . . . .

List of Figures

Illiol-l~"':;"I)"""''''''_''IOO('_IOOIJll Jl~"'--"")<IUI_ L.I0(._nlll ..

U s.-....-..., .."'l'l'I) IOOI • • IOO) 41 USlpoJpn>bobiliI' . . "'l'l'I) _ _ I!JIl(• • I:'O) 42

ftcun,<kn'o~;~I...._oo~I...\ono.FInt.,..'OIloloflllilt 0 6 , _ I ... Iof.wfllll, ...i'binl~b.M\I,~.I\h<honp

U Sipol~ityfor_dalapoint• •~pointfor....,pIo_

Chapter 1

Introduction

Quality bas become an important consumer decision factor. Even long-termCl~

to identify special causes and to bring the process under the influence0fa stable

system of chance causes. The basic philosophyofSPC is that a product coming out ofacontrolledprocesswillbeofgoodquality.The timely identification of changes

de,~lopcdbyDr.W.A.Shewartintbe1920s(seeShewart(1931)andShewnrt(1939))

by comparing the observed values with limits derived from past experience. Control

Commonly used Shewart cOlltrol charts areX - RorX -Scharts for mea.- sureddataandtheproportiondefective(P)chartfornttributedata.Control charts

troI charts are insensitive to subtle changes in the process (see Ryan(2000)) . More

islessthan1.5times the standard devintion. Cumulative sum (CUSUM) charts and

smallchallgesin the process (see Ryan(2000) and Hawkins and Olwell (1998))

1.1 CUSUM Chart

The cumulative sum (CUSUM) chart is widelYllsed for detecting small shiftsina process; it was proposed by Page (1954). To implement conventional CUSUM charts,

roeuson twokinds of process shifts: an upward shift in the mean(J-L'J>p.l) and a downward shift in the mean(Ill>P2), where IIIisthe process mean before the

illdepelldentlyandidenticallydistributed (iid) observations of size n from N(p,er'l).

S6=0

S;;= max(O,S;;_, +X.-k), n=I,2, ...

wherek=(jJ.J+I12}/2.[fS;i>h}wherehistbecriticalvalueandafullctiOllOrt'l andq2}then we conclude that&1lupwardshift in the process mean has occurred.For

So=O

S;;= min(O,S;;_, +X.+k),n~I,2,..

andq2inadvance. Another approach is to construct aself·starting CUSUM chart where we do not require these estimates in adV8Jlce. LetXjandsJbcthesample meanandsamplevarianceofthefirstjobservations.ComputetJ=R~

and for eachtjJdefineUj=([l-I[Tj _2(tj)],whereTj _2is the cumulative distribution function of the Student's t-djstribution with(j-2) degrees of freedom, and~is the standard normal cumulative distribution function. The random variableUjbas an exact standard normal distribution for all j>2 and theUJ's are statistically independent. As n increllSCs,./(j--t!-tl, Sn -G,andUn- t~' "N(O, t). We now

B;;=max(O.S;;_,+U.-k). n=1,2, ...

S;;=min(O,S;;_I+Un+k),n=t,2,

wherekisthe CUSUM referenceva!ue.st=0, and Sij =0. Leth",obethecntica1

in the process mean occurs whenS:>hn,o orS;<-h".o. Hawkins and Otwell (1998) have derived various combinations ofk and h",o in relation to tbe in-control average run length (ARL). Tbe CUSUM cbarts are not as efficient as Shewart controI

of Standards and Technology (NIST)jS8MATECH e-Handbook)

1.2 EWMA Chart

Theexpollentially-weightedmovingaverage(EWMA)chartisalsowidelyused for detecting smnll sltifts in the mean;itwas introduced by Roberts (1959). The

w,=J\XI+(l-).)Wt_l, t=1,2 , . where>' (0<AS 1) is Lhe weight, and tbeiuitial valueiswo=X.Xlis the

q~.=q'(~)[l-(l-A)"J

Hereq'lisestirnated8S~,WbereMRistheaverageofthemOVillgrnngesoforder

If,.\;::: 0.2, however, (1_,\)21 will be close to zero for allt~5, 50u;,can be

approximatedasa'l(6)·Thellthecolltrollimitscanbeapproximatedas

the process mean. EWMAchartsarealsonoteffectiveat identifyinglnrgeshifts They nrealso inefficient for detecting outliers (see NIST/SEMATECH e- Handbook)

has led researchers to develop efficient melhods for identifying changes in process

1.3 Change Point Problem

points. MethodssuchastheHawkinsmethod(thechangepointmodel)proposedby Hawkins, Qiuand I(ang (2003),themodified informationcriterion (MIC)byChen, Gupta and Pan (2006),and the Bayesian approach for the change point problemby Bansal, DuandHamadani (2008)nrecommonlyused. The Hawkins method assumes

ratiotcst and a modificatioll of Bayesian illformationcriterion (BIC). Here too the

(ovality),strength,etc. Consider the manufacturingofnn engine valve forautomo- biles. The circular runout (ovality) of the head-stock side of the valve is an importaut

runout. data is showninFig. 5.l (page 75) and it canbeseen that the runout distri-

The empirical likelihood (EL),propooedby Owen (1988,2001),hassimilarprOj>-

suggestedby Chen et aL (2006)i we replace the parametric likelihood by the EL

(2006) and Cben etal. (2008) to avoid thenonexistenceofsolutionswhencomputing

approach to change point problem by introducing EM test, proposed by Li,Chenand Marriott (2009)8nd Chen and Li (2009). We8pplynurprnposedmethodologiesto

rnthe next chaptec, we briefly discuss the existing approaches for the identification

webrieflyreviewEL,iutroduceourproposedEL-basedinformationcriteriou(ELIC)

Chapter 2

Review of Change Point Models

J(x;;Od for i= 1,2,3, ,7 /(xi;8:z) for i=r+l , In,

8ndf(,) denotes the probability density function, We asswne that the process has exhibited a change at thetimepointT,i,e"thein-controldistriblItionisf(:z:;8I)and after the change in theparamcter, the new distributionisJ(x;8'l),Inthisthesis, we focus on identifying the changes in the process mean(IJ)and assurne that the process variance(u²)isunchanged,Lc., we consider thecasewhereJ..&i'f:J..&'l and a?=a~=u'land the three parametersJ..& .. J..&2,anda'lare unknown. Moreover, we

the approaches proposed by Hawkinsetal. (2003) and Chen et al. (2006)

2.1 Hawkins Method

Hawkins et aJ. (2003) deveioped the change point model for process meanassum-

forealldafterthechangepoint(seeSenandSrivastava(1975),Hawkins(1977),and Worsiey(1979)), The data point with tbe most significant test statistic is identitied

II.n- ....- l ...~ _ Fw.,...-~,....'_

I S J . . ; · - l . _ ....--..~-

--"",., ....

... ---

^{. ...,.-}

__ ...J._~..t M _ ...~",_;,,,,~""Ioo_

_ . . . _t_poiI&j.T. . . .JI<I ...5 t _ '••_ _. .ait~.-2

_.-

1...p_tlpi/l<ao<elo¥tl •• _ _UI.I~'-...uvtofbtd_

f'r!T-.o>It,.JS:~PrfJT.. I>""1 -l.-IIPrfla-.oI>1t,.J

inequality is conservative. For this reasoo, Hawkinsctal. (2003)conducted a large

lhese are tabulaledinHawkinselal.(2003). Followmglhisapproach,whenevera

".,0"'''10,0(0.677+ 0.019Iog(a) +I-O~~~Og(a))

IV.~W'_I+((n-l)X.-S._.)'/[n(n-l)l. [llStcado[computingT,. [or each lS:i5n-l,computcrynasfollows

Ej.=(nSj-jS.)'/[nj(n-j)1 TheanalysisofvarianceidentityisgivenbyVj,.=W,.-E),.,leading toT],.= (n-2)E),./(Wn-Ej,.).Thencompute~u:,,.,thcm8Ximumof77,.,:i=l,2,. ..,n-l UT2I11&X.,.>h~.thecorrespondlngjisthe change point. The same j which maximizes theT},. alsomaximize5Ej ,..Therefore, ,,-'ecomputeoolyEmax.,.,then proceed to

T2mu:"n.oHawkins and Zarnba(2005) suggested a method for identifying the change

2.2 Modified Information Criterion The modified information criterion (MIC) proposedby Chen etaI.(2006)isa

functione,,(,,),theAkaikeinformationcriterion(AIC)isdefinootabe

AIC =-2l.(J1) +2dim(ii)

BIC~-2l.(ii)+dim(ii)log(n) wbereiJisthemaximumlikelihoodeslimateoflJanddim(X)istbedimensionofX.

I.Vo",,",J)-~""/(Z'.~')",,~,""/(Z""') ... ISjS.-l ... /(.) ...J_,"-'_TlooSIC ...

.... "" ...~_'.lI'l."'.J)...r - ... _,_w.,.._

BIC._ot . .~•

... _ o t. . . .,....~~_.;.,..Jl~2.-u.l»-1

_ _ '''_1 ...._~1.'''_---.......

Qoo ...(2OOIJ_._~I... _ . . - ·

JtIC1JI __1I'.(,;ol>.""'.Jl .. [2a.(jo,oI ..(¥~I)}•••I. J.I.2.

1;... _ _rlAL... ,....}. ...

where{1.maximizesl,.(fi,P.,n).We can select the model with a change point if M fC(n)>I~~~~_IM feu)and estimate the change point byfsuch that

MIC(f)=l,fJ~~_lMIC(j)

s"~MIC(n)-l,&~~l_lMIC(j)+dim(I')log(n) (2.1)

In this context,dim(J-L)=dim(ji,)=dim(jtlj) = d.UnderHo,Sn "'"X~in distribution asn~oo.IfS,.>x~,I_Q,whereQisthelevelofsignificance,thenthecorresponding data point is identified as the change point. Chen et al. (2006) showed the following

1.IfthereisachangeatT,T/nhaslimitin(O,l)asn~oo,thenSn~ooin

2. The estimatorfhasabestcollvcrgencerateofO,,(l),and via a random walk

tions is not easy to define. We propose an EL-based informationcriterion(ELIC)to

Chapter 3

Empirical-Likelihood-Based Information Criterion

Before introducing our EL-based informntiol1criterion (ELIC) for the changepoint

3.1 Empirical Likelihood

Bythecelltrallimittheorem,ifthcdatahaveafinitevarinnce,thcllthe sample aver- age follows the normaldistribulion and we can make valid iuferencesasymptotically.

However, efficiency maybecompromised. For example, assume that tbedata fol-

whereXis tbe sample mean. When n -00, ..com..rgcstc.17'0°IOhe,,.,iance.,fs' isa4(~+;),andbythenorma1itY6SSUmptionV8l"(s2)=~,sinceLhekurtosiS

willnotleadtoagoodeslimatefornVar(s2).ILwillsignificantlyaffccttheconfidence intervalofa2•Except for the normality assumpLion, it is not casyto 6ndadistri- blltionnl DSSumption that fits the data perfectly nnd allows both thcskcwnessand kurtosis to vary freely. To avoid this risk ofmisspccificatioD, nonparametricmeth·

ods can be used. EmpiricaJ likelihood (EL) is asystcmatic non parametric approach Lostntisticalnnnlysis , introduccdbyOwen (1988). ELprovidesa.data.-dctermined shape for cOllfidence rcgions; itc81lBSSimiiateknowncollstraintson parameteTS8l1d adjust for biased sampling schemes. lnthisthesis, \vedefineELbased on a set of estimnting cQllntions for thc paramcters of interest.

LetXI, X2 , •. _IX"bea set ofiid observations having a common distribution func-

onF.Tbe empirical distributionFn(x) is a good estimator of Fand \\"ecaDview it

F(x.l-F(x;-}~P(X.~x.}, i~1,2, ... ,n

Ln(Fl=)]£F(Xll-F(X;-}}

=)]P(X.=x.}

~)]p;

.. ,n,and

t,p;

Ln(F) cannot be a likelihood function in terms ofPiiftherenre ties inthe data To makeLn(F) a likelihood function, we can add a set of small and independent

lotheempiricaldistributionfunction,Fn. Since Ln(Fn)>Ln(F} for any Ff Pn, it is convenient to standardizeLn(F} by division by Ln(Fn). This con\lention leads to

R,,(p)=L,,(F) L.(F.)

ITp,

=(~-;n)'

=D(np,)

r.(F}=t,log(np,}

Profile Empirical Likelihood

functional oft.bepopulation distribution, say ¢=T(F).Infcrenceswillbemade

T(F}=,p.We must decide whichFbestrepre;ents¢.The concept of profile

satisfyingT(P)=,p. Theprofileempiricaliikeiihoodisdefinedtobe

L,,(II) = sup{L,,(P}IT(P)=11;FET}

,,_~_dim(X).dim(.J(o-a(IIl8ll.:IOOI)),UIlIll'I>oobow_. _ _

'boIOO(l-"J'llo"""fIdoncol_r...-.,

Optimization

S - ...

_01_...(...

l,.(FI-t,Joc'"

_~_~.",~O.;_l,t.,n,t,"'-l.ondt,MZ,,'l-O,Th ooive'hio_·_'ionproblem.l.opmce ..ul'ip6e<metbod • ...,. ..O<holfect....

... .-Itolil>d'bolW..-r)'poiotsolG(•. Al""ilh....-<"'.aodA,II'.obuoin .-nODdA ....beobuoiDO<1.,. ...n.... t,"'!riz".l-O.U...'lUolafr""P

11'(.) _-:!r.(.)-2t,locI1+~'I(X,,'Jl

Thisconvergestox~wbenn--+ • wberedistbedimensionofiP.

3.2 Two-Sample Problem Using Empirical Likeli- hood

problem using empirical likelibood (see Jing (1995) and Liu. Zou and Zbang(2008)).

butionsf(x;Jl)and f(YiJl-6). We wisbtotest tbe bypotbesis

Define 0=n,/n,wheren=nl+1121and assume that(J=nl/n- 400E (0,1) asn-t Let(PI,P2, ... ,PnB)a.nd(QhQ2, ...,Qn(I-O»)be probability veclorscor- ,Xfl \andY\, Yz, lYn~respectively such that8Pi=

E(x-Jl)=Ocorresponding to tbe first group ofobservatious and Eh(y'Jl)=E(y- J.l) =Ocorrcspondingto tbesecond group under null hypolhesis. The profile EL for

~,." ~

..

tM~,,~)-O. t.90~("'~)_O

U""",.. t.-""lt",u.hjpli<r"",<..>xI(..~iinSt«""'3.1_2),EL"uw<•

.._A_~,j_.1'2.

n(l-'HI +{l-I)'>'.h(".~))

f:~'","'.')

t,q,/l(IO,,.l-O

>.,+~,_0

(JI)

'.'lIol.-"'IoI:(""-~Io&I:.. r'.i,'f{""""

-...;I-II ...lo(.-fl.-t ...I ...II-II-'-'o... ,,), IU)

.,..._,.. ... £1._.- •. - ...

M'I'I.lIt ll_r'.i.'lb•.

",.t,

l"ioo_ \l_.-""'-_I _oI,.,

_

__

__

3.3 EL-Bascd Information Criterion

1'l',..."plrif..liu,hIK>O<IIn(2.1)_10II ....ut ..lm.'1..oq...i<><II.W...

In\<tootodIn'bothonpj~,bo_,,"Tbo_ploX,.,~.,. .,X"COllbo~

1O ... ,... ll'O"l"dMdodl'l"ltodo... pohI ..j.TIta_ll'O"P.-...~,_

'_ll,2. .J} . . . ...-.-_{j-+l.j-+2, . .~)

To_...£... ..., ..EUC._...,...pt'Ct«'dIwo.-.-:lio.llto_

....1lII:"'luationE91z,,"I_Elz,_"•• o,'_1,2,. ',J,OOfl'<llp(lOdinato'!>ollrtt

""pondl"ll,o,!>O"""""-lIfWPo(n, _ _ "",lefnnllhypOObeoOo,ff,,S.O ThoprofiloElior"ioloundt"._'-"ibDJ

p,,-nl1(I+,-,~.91z""n'^;·1,2,.,n,

• •"(I-'l{I+(I~tI'j"lh(Zl,,,n'

n..~_..oIt,pl;<o-o~.&t>d~on.:l.... _".,.tboaolu'iono((J,I),,;tbtho ...ima1OD&oquaI""'._""thio,''''' ...EL'..lor'''''''ionlor'bo . . . .

By considering the no-change-pointC8Seas weU as a change pointatj,wedefinethe

ELlC(j)=Wj(/L)-(~-l)'IOg(n) j~2,... ,n-2 (3.5) whereWj(/L) is defined in (3.4)

ELIC(T)=2~~_2ELIC(j)",X~,^(l8n-too.

IfELIC(T)>X~.I-O'then tbecorrespolldingj is the change point We know that tbe parametric likelihood ratio statistic has ax2 limitingdistribu- tion;thisisooeofthemajorpropertiesusedtoshowthatSn(2.I)forMICfollows a x3distributioo. Similarly, Owen (1988,2001) proved that tbe EL ratio statisticalso hasax2limitingdistribution. Variyath (2006) andVariyathetal. (2010) proved that theEL-based information criterion for variable selection also has aX'l approximation.

setofestimatingequations,anditcaneasilybesbowntbatELICfollowsthex'lwitb

imated byardistribution when we considerasample size (n=200) feasiblefor

MICbasedon lOOOOsimuJations. Jrthefalse-alarmrateisO.05,the95^thpercentUe shouldbe close toXi,O.9S=3.841459. However,forn=200ourquantilesforMlC and ELICare 7.821 and 8.138 respectively. Note that for larger sa.mple5izes(say n=1000),the95lhquantileofELICis3.9157,whichisclosetoxi,o.gs.However,in

on large number of simulations (detailsare given in Section3.6.1}. Since we did not u.sethex2npproximation, the theoretical proof is omitted here

3.4 Technical Problem and Adjusted Empirical Like- lihood

n2issmall.This is mainly becauseofthenone.'Cistenceofsolutions to {3.1} whenOis

_ _ _ ... ....-_01 ... __""_10 ... _ _

01 10EL,Md ..._DJC .... 1 ' 0 _ ....

... o4-cedELIA[LIIa...olI1lXJlll_h ..

01.1:1110). 1.0<. _"z,.~~

.

... _ - - .. AD..- - - _ -

... --... -

^{---- ...} -

.... _ ... _ ' - . _ _ 0 1 · '.--"'.1"100_

. . . _ ..( J . . I ) ~_ _-r- £UC bo _

""" ....~rmpiriallo&.IilooIiIIoodIoIll_1.,­

t:(F)-~"'",+"t,'.,.

•

• i!:O,I-J,2,

t'",.I. t'..l

t'--" t'...

flo-,.g.{J+~-'':,'g,j'^{'_1,1,_ .•nL+'}

"-n(l_i>'){J+('l_"1'.i, .... ),'·J·2.. ,,"'+J

" ...,._(~,+J)f(~+2).Tho~muh;pIitn~,aod~"""bo_...

'boooluu....of'boIoIIooriOf;O<lU&<ioPo

1i<(I')--"'-loc(",.l-"t,' ..,J+r-,;:,'g,j -n(l-r)loc(n(l-r)}-"t,'loc{l+(l-rl-'.i,'Iot)

(U)

(H)

-,., .",

1V;(I')-2!~loc(1+r-'.i.'!ll+~los(1+(l-'-r'i,'lotn (HI 1'ItiorotlYft'l<'llOx} " ... n_OO(a.." ..01. (2008)). Wouoo lV;(P) i""""'<lof lV,(I')iII(3.,';lIo<EUClo<dIaqopoiDt~..

ELlC(jI_II','(I'l_(~_II'loc(n)j_1 (U)

£Llc('l->s;"'t:-1ELlC(jl~~:. ~. ~-"'" (3.10)

II'..., _-loI:(~')f2ond .... _-1os(",)f2i.oorliDlll!a<IoD.udioo.Fo<

"""",<loI.aiIoOllAEL,m.rtoCh<D<to.l.(:lOO8)ondV"';yMh<to.l,(20IO).

3.5 Computational Algorithm

W".-d.be ••xhl>ed Newton·R.apMoaallOO'bmolCbon,Si''''''oad W.('IOO2}

to"""""..'be[L""Io ...IotI<Il_"(3.8).AeaI1didato~p<Ii>II.)dlvid..

oI",~)ODd'bel'ftDlliD.i... "-j_ioaolorm'bo _ _ ("""

Ii_" _^{g(',.~l-(z,-~) ~It, _.(Zl'~)}_(z,-oJ.n-oddlbe

additionaipointsg"'I+l=-g"I·a.,\ and h...,+l=-h..1*/lnltothefirstgroup and secolldgroup respectively. Represent (3.6) as

l

A= A2=E.q,h,+qn,+lhn,+I=O

I-I

A,=~I +~,=0 Compute the derivatives of A with respec:t to{J=(At,A2.Jl) as

[

D= ~ ~ ~

~~~

3.Usingtheewtoll-Raphson procedure, compute the itersth'e estimates of/1=

4.Chcckthecollvergence:ifl,Bk-,Bk+ll<=1O-5,tbcngOl.ostep6iotherwise

5.Feasibilitychecking:ifeitheroftbefollowingoccurred,set"(='Y!2,then

• min(p.} <0,i=l,2""l nl+1

• min(q,) <0,1=1,2,

6.ComputeWj(!") from(3.8) Bnd findELIC(j) from(3.9)

mum,ELIC(T).

3.6 Performance Analysis

3.6.1 Empirical Distribution of MIC and ELIC

whell there is no change point, thex2approx.imation for ELIC and MICdoes notfit

change point problems (see Cbenet al. (2006)). Wethereforecle<::idedtoconstruct

ELIC 6.4440478.1377212.423956 16.647301

To assess tbe ability ofeachmethod toide.l1tify thech811gepoint, wegeoerated data sets withn=200andachangein lhemean from the 101-(T= lOO}datapoint

areO,O.2,0.4,O.6,O.,and1.0.Forexample,ifT=lOOandtheshihisO.2,we generatedtbefirstlOOobservationsfromN(O,l)andtheremaininglOOobservstions rromN(O.2,1).

Foree.ch dataset generated, wecomputcdELlC(.,.) using (3.10) andSnusing (2.1). A change in the mean isdetectcd if these values are greater thal1 thecorre- spondingcriticalvaluesinTable3.1withtheappropriatefa1.se-alarmrates(a). The correspol1dingclata pointisthe change point for both ELiCandMIC. We repeated

change in the mean, weexpectthat tbesignal probability shouldbeequaltothe de.

.[2]...

'[l' .".---

I: / ,:l

J. /' ,. /1'

. ,;i . ) '

.,~

.

.. ..

.. ..

.. ..

..

f'i&u:'oJ.ldtpit"'bolOpol~'yiortbo_kllt.l\o<lolt"'~poU-.

.... 'bo-.nally<tio<,;\Ho<eddol.o,."""" ... _ ...ihodfrom<bel01~... ""'''' .._ ( I _...'bo<:hMp"","'io"IOO).F'roaI~J,J.... _tbo'","""ho

~". . . ."pwwdobiftl.'''"_.'booifnalprobobllit~I'''''-''t"iWl¥Jor boIh~UCand.\lIC.~,EUCil.. """, ..MIC"_'bo,i.t;UC, .. did

__ .._Ior ...l L.... _ _ - - . - . . .

... por-...:--.

w. ^.~iI. . _ . . . .IW·r,_IJIlI _ _

_ _ _ ,.. ... il._l50 ... _ ... _ _ "" . .l.lO

3.6.3 UnivariateExponentialData

data,weconsidcrexponentiallydistributeddatasetswithn=200and a change in the mean from the 1018t (T= 1(0) data poinlonwards. We considered a downwards

are 0,-0.1,-0.2,-0.3,-0.4, and-0.5(i.e., the mean changes from1to 0.5).For

exp{l) and the remaining 100 obscrvations from e.xp(O.8).Here,exp(p)means the

themeanatthe15pt('T=150)datapoint.

Fig. 3.3 thatthesignaJ probabilities for ELICandMICaresimilar,indicatingthat

"ull blpotbMlo. \\'b<n 'b<n 1o ....1fIill..-.,'bo~l<diloood...loIloo.n1l d11f«0011 brq{l+~l.~llotbopn;><wo""""boIoRoh!fl...d4Iotbo"'_t...ol obift.lfl"l,t...I~I:slUoil\l'ho'l\oylo<_""'*""'".lo&ll+~l _be~..~.whrlllo~bly...u.TbioIo ..byMIC ....

ow..p<rfoomaooo-tlh..,.,...,....d_loollikeli _ _Foo-'bo ...""""",lool

diotribo,;on("I'I"<"'imaliooo)oad'booxpo'l<,"'-ldiotribu'ioII.lo.'\lI.JA(whtnlo ...lll'n,»..,l.MICwi'h'bo""""*'diotriboAiooo(OWI'I"'l....ioalfailo"' ...ify

woll .. MIC.itb'bo'ruediotrihutJooo.1'hoo,wbon_DlIoop<oeI/y'bod;O'rlbu'... '"

which lDay lead to the useofanapproxirnaLedistribution in MIC, ELICperforms bettertban~IIC.This gives ELICaciearadvant8ge, smce we do not need to specify

Chapter 4

[

I

EM Test

4.1 Bayesian ApproachrorChange Point Problem

~'hlo_"'''ll'''';''''''''"t''''''LttX _X"X,

I(Z,il"P)b ._1.2,3._,r

•• Itotb_a..-a--'...,.~...

n.._""_...tobo ...

iIotloo----

-

[

U/lz",.,.,.1Ii1(',,.,,,.) ...-1.1.

£,1Jo, ...'O\>I- •

D,/lz",.,.,.I. ...-.

U, •.-...,tIoo

...,.IoV-""

Ilz ,..,,.. ..).t,.i.I/(rlr,,.,.,..,1'l

lIz",.,.... ,.. !!/1....,.,.".!!,I\',.,..,,.I.'-I.1.

f" " "'_1'~A - ..."'".J~._

~_-(.I_l.a..t,i>... ;... """ .... _ ..._/!rlr,,.,.,..,~

-~----"""I·_""

""I.{:::::

HereT#= n,sinceT=n impliesIII=112(no change in the mean).Hence,

L(ji"I'o'P,q:X)=(l-P)~q(T)f(XIT;ji"ji"P)+Pf(xln;jil'p) (4.1) distributionorq(;). Thema.'<imwnlikeliboodestimatororq(.)is

{

I if sup !?(ji"l'o,plx)= maxsup!?(ji"l'o,plx)

q(T)= I'1,p2.P 1~~"-1l'1,p2,p

O,otberwise

U;(ji"ji"plx)=(l-p)f(xiS;ji"I'"P)+Pf(xin;/."p) (4.2)

Bl\.nsalctal. (2008)used the expcetl1tion-moxilllization(EM)algorithm (Demp- ster, Laird and Rubin(1977»to estimate thechangepoint by maximizing L(Ilt.J.l2,P,q:x). TheE~'1algorithm isaniterative method which alternates be- tween perrormillg an expectation (E) step and a maximization (M)stcp. An£..step

I =_. ----

....

--..-_ ...

/.hoo._"Z,Tl.J,f(I..,..,.'1 lll-J')f(.l/,rir..,._",

To_ ... Jlzl.;/f';.!f'.iJI'l"'j<6I ••'_I_

1,2. .-I_/t."'.;.,:..P'l""fAl_I io,.I';.1>_t-'_ ...

_ . . . . j'6_<Jl ...~ _...-<Jl

._.,;,.,"'_;,!f'.1>.f"ioiP-b,.

Pl._.',ICjo'(',i4',p'.~_I(tU\z.r·,,..(^••l

1

(4..1)

"'1.»4,,,,,,,~_ _ ,, __, , · ~ < J l

...

. . ...-.-n.io ...!;f(.l/(ZIoI//(r,.I"~r.,-l)

.,.,. . . .pa;.I _ _ ...p..i1f!,j,!/'.F_f",,01

...

""~

EIloIL.lr.,.,.;.'/'.I1'_t"'I_jIk,.' ... /In..•,.',.., ..~ 1-,) ...onl.I .... /IZl" ...~'t-J'.,

1loo ...".)-I/I.-I'.-I.1".-I~. . _

... ...u._ ....^{tl'.~._1.1..• -l....}JO<l ....~".)_l._

"""II)_~_ ~'I')i~lz~)

~.Jl'\.) ~1"(.)JIt'IZIo) r...d>' ..-...aluloil::r'O'!I~_ 1/(z;~)_A(z)upjTlzl/< -..(,o)J tho

.

.

"'f(zll,,,,,"')-~A(Z;)+bT(Z')"''',~,T(Z')~'-I'(I',)-I'-IJ'v..) Tloou~_oli"t.,). .

*." ...,...

ts,.;.")II) 11-,IEf"'II)S,/U'Iz\!) .. "SJ"lzIooJ .·lIoll-~- "~,

~LjlIJ('1 (1_'I~t'll,)lJltI(z'I).. ,..jUIl0\001

lo'bo.-oI.""".,...u;c ...pt...Ior<J\,).... _ q ( . ) .... uniIo<m

binomioolpriorJcrq(.). • • l.2... ,n_I,oo.d",<aIlbo,.,lmat<dlfi'.lIllknoortt

... ,(0 ..'''l)''~''''''''~n..,,,,,,,~,,,,,.,,,,,,,r...,...~.~~

i~"'. ~('~l~(')^_~('~Jl(:=:)(i~)'-'(l-i"I)"'-lj((II('la)

(n-2)E';'~(') ^{(n_ 2)}

E(:

-, -, (U)

(1-'it~ll)t'I"'I+,sj-II~.1 tar - ":"

'1-'J~""i'Ii"(r\tI+"I"'(""

I:1.5"·S,lt'III/"'(OI

;.{'''_.~.".

- - -

~.-'l#"tI"'(o\Il -s,-t.~.-

ThIo~_ _ .. """,,,,,t&,ioMIly.,.poIIOl..olnookIDYOI_'boE.\1 p<O«<I_I" ... 'boEl.tprooo<Ul'O.q"I.. _ODdbnvlb'd<peodI""'he

""' ...~""_oltho~I.'Ioo. . .<>I ...l""'....ul<prIorb ....),o..-a ...{ D ) ! ) - - ' ....Iol_<>I....j_l!(._n._I.'l, .~

I,., ..._O; .._I._._D.U(b ...- . . l b ' . . - ... _

... __ 01..:.J ...s.,-~""_...~ot

... _-

'1.2 EM Test

TI> oI ...O;: . . . .t.o ...DB,_

Q,oo_ur...',...- ...D l _ . . . , . .. _ LooX"X.. ,X.bo •.--~_... /I .. ,.I... . . . -

'.l1,...I - t , ...W-IrJ!(r...,l .. "!( ...ll

..._'(OSflSIJ_I-f1 ...t1.e ...~-._,.,_"'...tbo .u.boI...-...·n.... ..~liIoeIlboodf__iaaio""lIC':llO""

PL"(lJ,I',.,,,)-t.(IJ."'.I'.)+'(1Il

"'.(I'-)-2fPL.f.l.;1ro,....} ,PL.(U,.... ""I} oJ

...pl.,,(I\,"I,"')iI~ I&Ad ....&AdPI.,,(ll.~,,'."')bu."'"".

""""' ... M.lt!o)boo. X'·,ype~ti,.t600ributloot~If_pt"'"I &Ad 1Idol1nodi"W ..ol.(2009)J_"",O&ti&fio,I. Ir't>o,totof<>llow.dill..-t<OO<ltl,

..'bonoolvtd'hilprobItmb!'upd... 'bo_ollJ"""&'boE.\l•...,'b...

ll~'boDl·olcori'bm,'hq"pdet.bo<h'... mi><i"l!pn>ponlootlJ&Ad'bolUW"l!

[>OI'OI""'....Ij.,.I',).Tboy_I'... E.\t .... toocllievebettoteftkitocy'ha.o ' .... olM.lt!o)b!'updot ...'bo ... olt\"Tboy_.n_ol;nUIoI_

....i<titEM;..I_,~/\I:..,ltlot))...Jio'... mllnl:>erol!""iaIw.l_ottlo ... "'io'bofbed ..._ o l l l...""'"-They~_~roo>dI,_tbot

W.~. . . . ' ...EM In' ...~~tolboclwtce[>Oi<:Jt problt"'"""""""b!'u.-l (~),F<Jr'boDt.... prooodw., .... diYi<lo 'bo ....plointQdilferentlJOC't>"(K)omdoet'll.)biV>fo<.-VOOpODdlowloroll 'bo~F""t>"&AdporlonDlboDt~",Ior.6nd_oIl...uc-

(m).Thedatapointwiththehighestq(s)valueisidentifiedastbecandidatechange

Eacb time, one group bas a highq(s)value and the others have lowq(s)values.Prom

with a large initial valueforq(s),withinafcw iterations it willhaveaq(s)va.lueclose

4.2.1 Step-by-Step EM Test Procedure for Change Point

1. Divide tbe dalainto Ksubgroups (Le., number of EM lests) and set tbenumber of iterations for each EM test tom (we considered 5 and 10)

2. Settheq(s) va.lueshighforaJlthedatapointsinonesubgroupand low for all points in the remaining subgroups such that theq(s)sum is one(s= 1,2, ... ,n-l)

rnaximumq(s) as a candidate ror the change point

4.3 SIMULATION STUDIES FOR NORMALLV DISTRIBUTED DATA (UNIVAIlIATE)54 5.Repeat steps 2-4Ktimes, and each time assign the highq(s) value to a different

6. Frolllthe/(candidatechangepoints, identify the data point withthe maximum likelihood value as the cbange point

Simulation Studies for Normally Distributed Data (Univariate)

4.3.1 Case 1: Data With No Change Point for Mean Inthe case of no change point (the null case), following Remark 2.1 of Bansal et aJ.(2008)wecheckedthevaJueofw(j)(n)using(4.3)atthefinaJiteration.lfthe value is greater than an appropriateconslantd,weconcludetbat thereisnochange point. No specific vnlue fordis providedby Bansaletal. (2008). Wesimulated

wW(t)vnlues. \VefoundtbatwUl(n)isthe second or third largest. Using EM test procedureswefoundthatw^Ul(n)iswithinthetop30%ofallw^U)(t)valuesafterm iterations(m=50r10)

To...". ... _ _ ~-.I.. -

.oI.lDJIllR1r....,. oI_._to. ... _ _ ...

_1')_1,,,'"~ " " _ " ' . Uw...

_ _ • _ _ .\-1.. 11 ... _ ( . - ' I _ · \ · U I \ ' ' '

_Dl_...

... 0 . . . " •• _ ... _ . . - ....2,1... _...

... _ , of_r-n-.

_Dl__

...~".)_o.t. _"' -.I _ _tl.)

Rw ....-.l0l._~tI.)_O_2ud~tl.)0. ""Ioia_ ...

...~tbotl.)...-~-I ' b t _ O l _••.-:Ildoto P<Jial

"_IIIod_ ..'''''"..,.;_O' ...t l . l _Rw _ _dwIp

1"""'....((IQIl'I'..'bolik<Iiboo<l ..lfl''''..cboafO ...n'.ond'''''<andIdotecll.onto I""" ",'h ''''' ...Intum lik<1iboo<l val.. ( _ ' b o K.:II-..) " ...'IIiod.'..

. . . P<Jial.\\·.p«bmodlOO'Xl.m.u.., _ ...lploo... tboptOC>Ortl<Jollol'_

_ .... poIaI"-..Iiod ..tbo . . . . .pai fbf.-pow\ooo._ .,...,....

... _ _~koIBoo.1..ol.fDlII) \\ . - - . t _ .. ""

_(Iudl.~) ~_ _,p- ..n...I.U... U .... - " " _

4.3 SIMUI..ATION STUDIES FOItNORMALLV DISTRIBUTED DATA (UNIVAIUATE)56

similarresults to thefullEM algorithm (Bansaletal.(2008)}for identifying the

Wealsoconsider a larger sample size, n=30,withchangepoints(7")at6,15,

4.3.3 Case 3: Binomial Priorforq(s)

4.3 SIMULATION STUDIES FOR NORMALLY DISTRIBUTED DATA (UNIVARlATE)57

suggest a binomial priorin(4.4),and the iterat.ivesolulion for lbemaximumLikelihood estimator of,.isgiven by (4.5). Tbeestimatesof8land828reobtainedtbroughtbe procedurediscussedearliere.xcept[orq(m1(s},whicbisreplacedbyq(s;i(m1).lnlbe

and,foreach prior, identified the candidate change point as ni(m) where...,(m) is the

each ca.ndidalechange point witbq(n.y(m) =1.Tbeshift in the meanisset to1.

asample izeofn= LOwith change poinls at 3, 5, and 8. Bansalctal. (2008) did

Wealsoconducted simulation studies for a larger sample size, n= 3O,withchange pointsat6,15,and24. We simulated data from a normal distribution with a unit shih in mean at the change points. Asummaryoftheresultsisgiveninf'ig. 4.8. \Ve see from Fig.4.8thattheE~'ltestperforms8Swell8StheEMwitbfulliteration

Performance of MIC, ELIC, and Bayesian ap- proaches

Wecomparoo tbeperformanceofthe Bayesi8D approach for the change point.

problem with MIC and ELiC. We restricted our comparison to the nonparametric- prior case in botb the Bayesian approach and the EM tests. Since there is nowell- defincd procedure in the Bayesian approach tosignalthcchangepoinL,weidentified thepositionwherethechangepointwasdetectedandcompareditwiththntofMIC and ELIC. [<'or MIC and ELIC, we identified the data point correspondillg to the teststatisticsSn8IldELIC(r)and used that for the comparison. We compute the mean and standard deviation of the identified change points for thecomparison. We considered sample size n= 200 and two locatiollsof the change poiutsasper thc performance studies carried out in Chapter 3. In thefirst.scenario, wegcncrated the first 100 observations fromN(O,1) and tbe remaining 100 observations fromN(P,I).

subgroupstoK=5. Summary statistics based on 5000 simulations are given in Table

change point. Note that the standard deviation ror ELICand MICisslightly lower

ShiftinMean

0.2 0.4 0.6 0.8

100.17100.28100.14100.08 99.99

24.0214.378.9 5.69

on SOOOsimulations are gi\--en in Table4.2. Herealsoallthemcthodsidentifiedtbe change point correctly. Note that the standard deviation for ELICislow compared

0.2 0.4 0.6 0.8

116.8 136.49144.n14.W149.15

BayesianEMtest,m~5 ~Iean 111.66133.32144.7814.46149.55

generated fromexp(l),whereexp(l) isanexponcntiaJ distribution with meanl,and the remaining 100 observations fromexp(p),Le.,datafrom an exponentialdistribu-

46.87

41.2335.3228.69

112.91123.56133.09138.78 45.4838.51

f :ll

II ^f H,

f ~ll ^II

^f H

^II

II

f H,

II

f :t

^II

Figure 4.1: Signal probability for each dalapoint as a change point for sample size 10. FirstfigureineachpanelreferstoBansaJetal.procedure;secondfigurerefen

I=j _;!!!!!!!!! I rj _'!!!!!!!!! ^I

.~11'SiPp<o/>oI>iliI11ot_dalapoUc".~poIao"'-pM­

10n.. ...III_.-oI . . . " ' - ...~_&c-_

1001 6 _ , ,...lorolNAU.....-I,... ... _

IO'_ ... Ior_Uwll.lo ,.... ..~...

I a::::: _~ ^{! J I} a

^! ~ !

Figure 4.3: Signal probability for ea.ch data point asBchange poilitfor sampie size 10.F'irst figureineachpanel refers to Bsnsaletal. proccdure; second figure refers lOEM testbased on 5 iterations. First panel is for shift O.5,secondpanelforshllt 1.0, and thircl panel Cor shift 1.5 with change point at thposition.

FlJw'ol~.Sipal"'-"Oli<,""_dooI et.ap_b-....plo . .

30l'iIot ...i I I _ ~

....

lOot _ _ . .~ _fln<puollobolllAIU, Ior_

Flpoo~~:SlpalprOOobllit,Ior-adll"'poIoo • •~poIoolor.-mplo'"

3O.f1<ot -pooooI ~ ~-~IIpe-

... Ol - ' _ i _ r-. Ior . .U,~-'Ior^{. .}

1,..uol.- ...I o r _ U _ ,... ..I~...-

f1c...46, !llpo.Ipo<>booblt!,'bMdl ... M_ ... b JlIt . .

30f'\nI. . . lo_puol ...."'_.a1.~...,_

...

':11!!!!!!!!I':l!!!!!!lp

f1pro4.J'SIP~YIo<_~d&l.opOlol··~poIaolo<-'-"

lo.nr. ... _I"""'l ..."'t..n_kol ...O I _ . - ...

1001 _ _ . 6 _Finlpooool ... _ .. 2. _ _ iIpN

JOF'Int ... _ .....,NII_"'...OI_,-..:-lllpn ...

.. Dl _ _ . . 6 _ _ f"d Io ~,.,....6 , _ i p N

... poIo& ..I~_Wtd ,.,.. ..21SWl.Io_..,1

Chapter 5

Case Studies

lntbischapterweapplytbemethodsdevelopedinthistbesisloLwodatasets. We donotknowthctruechangepointsforthcsedatasels,butouranalysiswiUhelpus

5.1 Example 1: Runout data

Int.hissection, we discuss the identification of tbc change point for thecircular runout (ovality) dataset discussed in Chapter J. ThcquaJitycharacleristicofinterest is the mnoulon the bead-stock side of an engine valve. The lower and upper limits on the runout, set by tbe customer, are zero microns and 0.5 microns respective1y.

hislogromofthehistoricaldalaofo\'ality(seeFig.5.1)indicaleslhalitsdistribution isnon-normal (slightly right-skewed).An empirical likelihood ba.sedon aconrrol chanisproposedLOmonitor this process (see Variyatb (2010». inee the small

WeapplicdalJfivechange-pointmclhods,lbechangepointmodel,MIC,ELIC, theBayesianapproach,MdtbeBayesianapproochwiththeEMtcst(m=5). All

5.2 Example 2: Chemical data

inBox, Jenkins and Reinsel (1994). A plot of this data set (number ofobserva- tions=197) is given in Fig. 5.4,and a histogram is given in Fig. 5.3. The histogram clearly indjcatcsthat tbedistribution jsnoo-normal (slightly right-skewed). From

Fig. 5.4, we suspect that there are multiplecbange points, around100and ISO Since it is not. easy to approximate the distribution of the concentration readings, ELICmaybethe most. appropriate method to identify thecbange point.

Weapp1iedallfivemethods(Bayesian,BayesianwithE~ltestandm=5,Hawkins method,MIC,ELlC)todetectchnngepoints. All the methods except ELICconclude that there is a shift at the 170tb position. Only EL1C idelltified the early shift at thelSOlbposition.Wbenv.-echeckedlhetest5tatisticsrorallthedatapoints,we realized there is also a shift atdata point 55. However, the maximum values of the test st.atistics occur only at the lSOUtand 1700'positioos,andonlyELICdetocted

F._U-I'lo<uI'tII•.,,,,d....

- I

Chapter 6

Concluding Remarks

processcandiminisbthequality. To detect a shiftillthe process mean (the change point), practitioners usually prefer conventional methods suchI.\SShcwart control

shifts. Better methods are needed for change point analysis. Hawkinscta1.(2003)

Sludent'slslatistic. Chen et al. (2006) suggested a modified information criterion (MIC) bascdon the likelihood ratio test and Bayesian information criterion (BlC)

However, these methods rely on parametric assumptions for lhequalitycharacteris- tics. Uwe are unsure about thedislributional properties of the quaJity charscteristic,

\veusuallypreferanormalmodelassumption. However, it can lead to an inoorrect conclusion. We lherefore need to develop a nonparametric procedure for change point analysis.\VeproposeanEL-basedinforrnationcritcrion(ELIC)toidentifychanges in the process mean. ThemainadvantageofELICisthatwedonotncedtospecify tbedistributionofthequalitycharacterislic. Using theadjusLed empirical likelihood, we developed a computational algorithm to compute ELiC. Our simulation studies clearly indicate that ELlC performs as v.'eU astheparametric-ba.sed methods,iftbe distribution of tbe quality characteristic is known. \Vbenthedistributionismisspec- ified or approximated, parametricmetbodsfailcd todetecttbecbangedetected by

In Lhe context of change point analysis, Bansal eLal. (2008) proposed asemi- Bayesian approach with the EM algorithm. This method is computationally expensive because of the use of the EM algorithm. ToacceleratethisBayesianapproacb,we suggest using the 8M test proposedby Liet.J. (2009) and Chen and Li (2009). We usedtheE~ttest with a specific initial value of the prior probability q to accelerate the identification oftbechange point.Under a binomial-prior assumption for q,

\\-'emodifiedtheBayesianapproachbyusingdifferentbinomial-priorassumptions.ln

bol.hcases, weusedthe EM test with afL"(ednumber of iterations, m=5. Simulation studiesshowthattheE~1test of the Bayesian aPPf06Ch works as well astbe Bayesian

\\'ecompared the perfonnanceoftbe Bayesianapproach,the Bayesian approach

parcdtootber metbodsinbothspecified and misspecified scenarios fOTtbedistribu- tional assumptioo oftbe quality characteristic. Weappliedourproposedmetbods to rea1 case studies, the circular runout (o\'ality) dataset and the chemica1 process concentration data8vailable in Boxetal.(1994)

weprcsentcd the simulation study for univnriate proccssmeanonly. We would like

qunlitychamctcristicllSwell. Inthisthesis1\Vcconsideredthcsitlllltiollwherethere is only ollcchangc poillt. But in pra.ctise, we may come across situationswherethere arc lI1ultiple change points (see chemical dataset discussed in Chapter 5).Sowe

Bibliography

11J Bansal, N.K., Du, Ii., and Hamedani, G.G. (2008).AnApplication of EM AI·

gorilhm to a Change-Point Problem.Communacotlons in Stats.staa: Theory and

[2JBox, E.P.G., Jenkins, G.M., and Reinsel, G.C.(199~).Time Series Anal)'si.

13J Cben, J., Gupta, AX, and Pan, J. (2006).lnformalion Criterion and Cbange Point Problem for Regular Models.Sankhya: The IndianJournmof Statistics,

{4] Chen,J.andLi,P. (2009). Hypothesis Test for Normo.l Mixture Models: The EMapproacil. TIle Annals a/Statistics, 37,2523·2542

15J Chen, J., Sitler, R.R., and Wu, C. (2002). Using Empirical Likelihood~Ieth·

odsto Obtain Range Restricted Weights in Regression Estimators (orSunoeys.