and Smoothers from a State-Space Model with the Feature of Time Dependent Dimension
by
c Wen Teng
A thesis submitted to the School of Graduate Stud- ies in partial fulfillment of the requirements for the degree of Master of Science.
Department of Mathematics and Statistics Memorial University
September 2016
St. John’s, Newfoundland and Labrador, Canada
Abstract
This thesis illustrates two approaches for the evaluation of forecasting, filtering and smoothing from a flexible state-space model. Parameters of this model can be time dependent and the dimension of its state or observed vectors can vary over time. The first approach consists of establishing an algorithm based on the Kalman filter and Kalman smoother as well as properties derived from the model. Another approach is to reconstruct the model. In addition, an extension of the model is proposed.
ii
Acknowledgements
I am very grateful to have the chance of studying at Memorial University and should thank many people for their help.
At first, I would like to express my sincere gratitude to my supervisor Dr. J Concepci´ on Loredo-Osti for his patience, motivation and continuous support during my master’s study. His guidance helped a lot in writing of this thesis.
Besides my supervisor, I also would like to thank the faculty and staff at Depart- ment of Mathematics and Statistics. My deep thanks go to Dr. Zhaozhi Fan for his excellent course, help and brilliant advices regarding my study and career. Taking courses of Dr. Alwell Oyet and Dr. Hong Wang is also important for me, it leaves me a wonderful memory. Many other faculty or staff give me support and encouragement in different ways, thank you all so much.
At last, I want to thank those friends met in St. John’s for participation in my life.
iii
Table of contents
Title page i
Abstract ii
Acknowledgements iii
Table of contents iv
1 Introduction 1
2 Preliminary knowledge of the state-space model 3
2.1 The state-space model and representation . . . . 4
2.2 Forecasting, filtering and smoothing . . . . 8
2.2.1 Kalman filter . . . 10
2.2.2 Kalman smoother . . . 13
3 Discussion of the joint distribution of estimators 15 3.1 Recursive method . . . 17
3.2 Model reconstruction method . . . 27
3.3 Extension of the state-space model . . . 37
4 Summary and future work 42
Bibliography 44
iv
Chapter 1 Introduction
This thesis would mainly explore the relationship among estimators like forecasts, filters and smoothers of a state-space model. We will propose a flexible state-space model, in which any subset of parameters can be time dependent. In addition, the dimension of the state and observed vectors of our model could vary over time, which is only assumed in limited literature, for instance, the article of Mclauchlan and Murray [1996]. The relationship among estimators could be well described by their joint distribution. Based on the algorithms - Kalman filter and Kalman smoother, we adopt two methods to obtain the complete information of the distribution. One method is established by researching on some properties derived from the model, the other one can be accomplished by reconstructing the model, which is a classic approach in the study of state-space model (see Anderson and Moore [2012]). Besides these, inspired by the work of Qian [2014], we extend our state-space model into a new one, and propose a procedure to calculate forecasts, filters and smoothers of the new model through the model reconstruction method.
The rest of the thesis is organized as follows. Chapter 2 provides an introduction
to our state-space model and preliminary concepts associated with it as well as a brief
introduction to Kalman filter and Kalman smoother. The organization and content
of the second chapter is referred to the books of Shumway and Stoffer [2010] as well
as Hamilton [1994], however we should notice that our model differs from theirs due
to the time-varying dimension of the state and observed vectors. Chapter 3 illustrates
the main work of this thesis. In the first section of chapter 3, an algorithm has been
established to assist us in studying the joint distribution, and in the second section
of it, we examine the model reconstruction method. we also extend the model in the
third section. Chapter 4 gives a summary and indicate our further work.
Chapter 2
Preliminary knowledge of the state-space model
Starting with the breakthrough papers of Kalman [1960] as well as Kalman and Bucy [1961], the state-space model has been widely applied in many fields such as statis- tics, economics, engineering and medicine. Harvey [1990], Hamilton [1994], Tsay [2010], Durbin and Koopman [2012] and Tsay [2014] present its theory and applica- tions in time series analysis. The article of Basdevant [2003] contains an application on macroeconomics, Mergner [2009] addresses applications in the area of finance and Jones [1984] provides an example of applications on health science. In order to solve practical problems, building a good state-space model is essential. We should know how to cast a structural model into an appropriate state-space form. The representa- tion is not unique, for one can enlarge the state vector but describe the same process.
The primary and standard tools for analyzing a state-space model are filtering
and smoothing algorithms. Filtering provides an estimate of the state vector at a
given time point conditional on the information observed up to that point. Smoothing
enables us to estimate the state vector at any time given all the available observations.
The best known filtering and smoothing algorithms are Kalman filter and Kalman smoother, that could be found in a large amount of literature, for example, books of Shumway and Stoffer [2010], Anderson and Moore [2012], Brockwell and Davis [2013].
In the first section of this chapter, we describe a state-space model which is the focus of this thesis, and give an instance to illustrate the state-space representation. In the second section, we would discuss the forecasting, filtering and smoothing problems and introduce the solution - the Kalman filter and Kalman smoother algorithms.
2.1 The state-space model and representation
Many linear dynamic systems can be written in a state-space form. Before explor- ing the state-space representation, we consider a simple dynamic system: first-order autoregression
z t = φz t−1 + ε t , (2.1)
where ε t ∼ i.i.d. N (0, σ 2 ). Since the future values {z t+1 , z t+2 , · · · } of this process only depend on the present value z t , we can easily to analyze the dynamics and make forecasts for this process. For example, we solve equation (2.1) by recursive substitution:
z t+m = φz t+m−1 + ε t+m
= φ(φz t+m−2 + ε t+m−1 ) + ε t+m
= φ m z t + φ m−1 ε t+1 + φ m−2 ε t+2 + · · · + φε t+m−1 + ε t+m
(2.2)
for m = 1, 2, · · · , which implies an optimal (in a mean square error sense) m-ahead forecast
E(z t+m |z t , z t−1 , · · · ) = φ m z t .
In order to represent a more complicated linear dynamic system by the state-space model, we should derive the dynamics of the system from an (n t ×1) observation vector y t . The system dynamics is expressed by the change of a possibly unobserved (r t × 1) vector x t known as the state vector of the system. The dynamics of a linear system can be illustrated as a generalization of (2.1):
x t = g t + F t x t−1 + v t , (2.3)
where F t denotes an (r t × r t−1 ) matrix, g t is an (r t × 1) predetermined vector, the (r t ×1) vector v t is taken to be N (0, Q t ) and {v t } ∞ t=1 is an independent sequence. State vector x 0 is assumed to be the initial value of the state, furthermore, we assume that it is N (γ, O) and independent of {v t } ∞ t=1 . We usually name (2.3) the state equation.
Note that 0 in the last paragraph symbolizes a zero matrix fitted its position, for instance, population mean 0 for vector v t is an (r t × 1) zero matrix. In the following context, symbol 0 would be used like this way for convenience. Similarly, Q t and O in the last paragraph are covariance matrices fitted their positions.
Like (2.2), we can write
x t+m = ˜ F t+m m x t + ˜ F t+m m−1 g t+1 + ˜ F t+m m−2 g t+2 + · · · + ˜ F t+m 1 g t+m−1 + g t+m
+ ˜ F t+m m−1 v t+1 + ˜ F t+m m−2 v t+2 + · · · + ˜ F t+m 1 v t+m−1 + v t+m
(2.4)
for m = 1, 2, · · · , where
F ˜ t+m n = F t+m × F t+m−1 × · · · × F t+m−(n−1)
for n = 1, 2, · · · , m. Thus, the optimal m-ahead forecast can be written as E(x t+m |x t , x t−1 , · · · )
= ˜ F t+m m x t + ˜ F t+m m−1 g t+1 + ˜ F t+m m−2 g t+2 + · · · + ˜ F t+m 1 g t+m−1 + g t+m .
(2.5)
Assume that the observation vectors are related to the state vectors through the equation
y t = a t + H t x t + w t , (2.6)
where y t is an (n t ×1) vector representing the observation of the system at time t, H t is an (n t × r t ) matrix of coefficients, and w t is an (n t × 1) vector which could be thought as measurement error; w t is presumed to be N (0, R t ) (R t is covariance matrix fitted its position), {w t } ∞ t=1 is an independent sequence and also independent of {v t } ∞ t=1 as well as the initial value of state x 0 . Equation (2.6) also includes a t , an (n t × 1) observed or predetermined vector. For example, a t could include the information of lagged values of y. We usually call (2.6) the observation equation of the system.
The state equation (2.3) and observation equation (2.6) constitute a state-space representation for the dynamic system of y. In this paper, we would focus on the system described by (2.3) and (2.6).
Note that the dimension of both state and observed vectors of this model can
change over time. The time-varying dimension of observed vectors has been well
understood and practiced. For instance, if some elements of an observation have
missed, then its size is accordingly reduced. If the observation itself has missed, the
updating step of the Kalman filter would be skipped (see Jones [1980], Harvey and
Pierse [1984]). However the time-varying dimension of state vectors had not been
enough appreciated until recently. Jungbacker et al. [2011] put common factors and
idiosyncratic disturbances corresponding to missing data in the state vector when
consider a factor model with missing data, then the state vector varies in dimension over time due to the variation in the amount of missing data. In our model, the dimension of state vectors is allowed to vary over time, not only for the generality of the model but also for the convenience to introduce the model reconstruction method in next chapter.
Because a t is deterministic, the state vector x t and measurement error w t contain everything in the past which is relevant for the future values of y,
E(y t+m |x t , x t−1 , · · · , y t , y t−1 , · · · )
= E(a t+m + H t x t+m + w t+m |x t , x t−1 , · · · , y t , y t−1 , · · · )
= a t+m + H t E(x t+m |x t , x t−1 , · · · , y t , y t−1 , · · · )
= a t+m + H t E(x t+m |x t , x t−1 , · · · ),
(2.7)
where E(x t+m |x t , x t−1 , · · · ) can be obtained from (2.5)
We take a pth order autoregression as a simple example of a system which can be written in state-space form,
y t − µ = φ 1 (y t−1 − µ) + φ 2 (y t−2 − µ) + · · · + φ p (y t−p − µ) + ε t . (2.8)
Then we write (2.8) as
y t − µ y t−1 − µ
.. . y t−p+1 − µ
=
φ 1 φ 2 · · · φ p−1 φ p
1 0 · · · 0 0
0 1 · · · 0 0
.. . .. . · · · .. . .. .
0 0 · · · 1 0
y t−1 − µ y t−2 − µ
.. . y t−p − µ
+
ε t
0 .. . 0
. (2.9)
The first row of (2.9) indicates (2.8) and other rows simply state the identity. We use the following notations:
x t def = (y t − µ, y t−1 − µ, · · · , y t−p+1 − µ) 0 ,
v t def = (ε t , 0, · · · , 0) 0 ,
F def =
φ 1 φ 2 · · · φ p−1 φ p
1 0 · · · 0 0
0 1 · · · 0 0
.. . .. . · · · .. . .. .
0 0 · · · 1 0
,
H def = (1, 0, · · · , 0) 0 . Hence equation (2.9) can be written as
x t = F x t−1 + v t ,
which is the state equation. The observation equation is
y t = µ + Hx t .
2.2 Forecasting, filtering and smoothing
The relevant state-space model is taken as previously mentioned:
• state equation
x t = g t + F t x t−1 + v t x 0 ∼ N (γ, O)
v t ∼ N (0, Q t ),
(2.10)
• observation equation
y t = a t + H t x t + w t w t ∼ N (0, R t ),
(2.11)
for t = 1, 2, · · · . Note that all the elements of g t , a t , F t , H t , γ, O, Q t and R t are known with certainty, and {v t }, {w t } as well as x 0 are mutually independent.
In practice, the main aim for the analysis of a state-space model is to provide estimators for the underlying unobserved state x t on the basis of observations Y s = {y s , y s−1 , · · · , y 1 }. When s < t, the problem is called forecasting; when s = t, it is called filtering; when s > t, it is called smoothing. Besides the estimators, we also want to measure their precision. The solution to these problems can be accomplished by Kalman filter and Kalman smoother.
For convenience, we will use the following notations:
x s t def = E(x t |Y s ),
P t s
11,t ,s
22def = E{(x t
1− x s t
11)(x t
2− x s t
22) 0 } and
P t s
1
