Time Series Regression

BIBLIOGRAPHY

Consider two random variables, y and x . A regression of y on x is a model of the mean (or average) of y, conditional on values of x . It is thus a common statistical tool for analyzing how x might influence y . If a sample of values of y and x is observed in sequence over a period of time, this model is called a time series regression .

Time series regressions are distinct from cross-sectional regressions, in which observed values vary across similar units at a point in, or averaged across, time. For example, a researcher interested in the effect of income on consumer expenditures might rely on a cross-sectional regression model of expenditures of individual households during the year 2006 on household income during that same year. On the other hand, other researchers interested in overall consumer behavior might utilize a time series regression of aggregate consumer expenditures and income as they vary from year to year.

The distinguishing aspect of time series regression models is the common presence of serial dependence —a correspondence of values at different points in time— which does not typically arise in cross-sectional data. In time series regressions, there is a sense in which the order of the observations matters. Indeed, one of the important functions of time series regressions is to estimate and characterize this dependence on time and to determine how different variables fluctuate together over time. Serial dependence presents practical and conceptual problems that typically do not occur in cross-sectional applications.

Regression models of the mean of a random variable trace back to Francis Galton’s 1877 work on the hereditary effects of parent sweet pea seeds on their offspring. The first person to apply regression analysis to economic data is most likely Udny Yule (1895), who investigated the effect of types of government relief on pauperism in England in the late nineteenth century using cross-section data. Time series regressions and correlation analysis in economics begin to appear at about the same time in later work by Yule (1899) and in important studies by Reginald Hooker (1901) and John Norton (1902). This research grappled with many unique issues arising in time series regressions, such as trends and time lag effects (Klein 1997, chapter 9).

In a time series regression of y on x, the random variable y is called the dependent variable, because the model shows how its mean depends on the vector of k regressors, or independent or explanatory variables, x. Formally, we may write the regression model as

where β is a k X 1 vector of constant parameters or coefficients (independent of time), e is a random error with mean zero and variance a , and the sample period is assumed to run from period t to period T. The subscript t on the random variables denotes a particular point in time or period of time, so it is understood that these random variables vary over time. Since there are k explanatory variables, the model is in general a multiple regression model. We interpret any particular β coefficient as measuring the marginal effect of a change in the corresponding explanatory variable at time ton the expected value of y at time t, holding all other explanatory variables constant.

The model in effect decomposes the random variable y into its mean conditional on x and a random error. The conditional mean is given by E(yǀx_t ) = β’x + E(e ǀx) = βx_t, in light of the zero mean assumption of e , and is assumed to be linear in the explanatory variables. If y and x are jointly normally distributed, this linearity assumption will hold exactly. Linearity implies that the marginal effects of the explanatory variables are independent of the values of these variables.

Given a sample of time series data on y and x, the regression in (1) can be estimated. Estimation assigns specific values to the unknown parameters to identify the specific probability distribution that most likely generated this sample. Once the regression has been estimated, it can be used for making inference about theory, for forecasting, or for understanding the effects of policy actions.

The key assumption of the regression model is that the expected value of the error term is independent of the explanatory variables

which is assumed to hold for all t in the sample. This assumption is called the strict exogeneity of the vector x. It implies that the systematic relationship between y and x is fully captured by the model for conditional mean, so that deviations of y from this mean are purely random. It follows that under strict exogeneity the error term is orthogonal to each of the regressors (the expected value of their product is zero) and that the covariances between the error and each of the regressors is zero. It also implies that least squares estimates of β are unbiased in small samples—no matter the size of the sample, the expected value of this estimator of β is equal to its true value.

A common case in which strict exogeneity does not hold is the autoregressive model, in which the conditional mean of y depends on its past values (for example, x_t = y_{t – 1}). It should be clear that E (ε_{t – 1} / x_t ) = E (ε_{t – 1}/y_{t – 1}) does not equal zero in this case, as required by strict exogeneity. For such lagged dependent variable models, least squares estimators are biased in small samples. However, they will be unbiased in large samples, even when strict exogeneity does not hold if the regressors are predetermined : E (x_it ε_i ) = 0 for all i = 1, …, k .

Both strict exogeneity and predeterminedness often break down in applications in the social sciences because of simultaneity bias . Such bias in estimating a time series regression occurs when the explanatory variable is influenced by the dependent variable or when common factors jointly affect both dependent and independent variables. Instrumental variables methods are a common approach to dealing with this problem; an instrument for x is a variable related to the dependent variable y only through its association with the explanatory variable x . In general, the appropriate use of theory to guide the selection of instruments and other identifying restrictions is essential to solving the simultaneity problem.

Another important assumption of the basic time series regression model is that the error term is serially uncorrelated : E ε_t ε_s = 0 for all t ≠ s . This condition means that past observations of the error term do not help forecast future values. If this condition holds (along with strict exogeneity), the least squares estimator of β is efficient in the sense that, of all the linear, unbiased estimators of β, the least squares estimator has the lowest sampling variance. If the error term is serially correlated, least squares estimators of β will remain unbiased as long as x is strictly exogenous, but will be inefficient. Generalized least squares methods allow efficient estimation of β in the presence of serial correlation.

The autoregressive model with lagged dependent variables noted above is an example of a more general family of time series regression models that can capture the complex dynamic interactions that typically characterize time series data in the social sciences. A dynamic regression, sometimes called a distributed lag model, is given by

Note that this model specifically accounts for serial correlation in the dependent variable through the a coefficients, while allowing direct dynamic interactions with x through the b coefficients. For example, the coefficient b ₃ measures the marginal effect (holding all other variables at all other times fixed) of a small change in the value of x three periods ago on the current value of the dependent variable.

Many time series variables wander over time without an apparent tendency to revert to mean. A dynamic process exhibiting this type of behavior is called a stochastic trend process, an example of which is the random walk . The best predictor of the current value of a random walk process is last period’s value; the change in a random walk is unpredictable. Time series regressions that involve random walks, or more generally stochastic trends, may lead to invalid results and inappropriate inference.

Suppose that y and x each follow a random walk. As shown by Granger and Newbold (1974), a simple time series regression of y_t on x_t will reveal a strong correspondence between the two variables, even if they are, in fact, unrelated. That is, the estimated relationship between the two independent random walks will be spurious. The common solution to this spurious regression problem in the presence of stochastic trends is to either (1) estimate the time series regression after transforming the data into first-differences (i.e., instead of regressing y_t on x_t, regress (y_t – y_{t – 1}) on (x_t – x_{t – 1}); or (2) regress y_t on x_t, but include lagged y and lagged x as explanatory variables.

SEE ALSO Autoregressive Models; Regression

BIBLIOGRAPHY

Galton, Francis. 1877. Typical Laws of Heredity. Proceedings of the Royal Institution 8: 282–301.

Granger, Clive W. J., and Paul Newbold. 1974. Spurious Regressions in Econometrics. Journal of Econometrics 2: 111–120.

Hamilton, James D. 1994. Time Series Analysis . Princeton, NJ: Princeton University Press.

Harvey, Andrew. 1990. The Econometric Analysis of Time Series. Cambridge, MA: MIT Press.

Hooker, Reginald. 1901. Correlation of the Marriage Rate with Trade. Journal of the Royal Statistical Society 64: 485–492.

Klein, Judy L. 1997. Statistical Visions in Time: A History of Time Series Analysis, 1662–1938 . Cambridge, U.K.: Cambridge University Press.

Norton, John. 1902. Statistical Studies in the New York Money Market . New York: Macmillan.

Yule, Udny. 1895. On the Correlation of Total Pauperism with Proportion of Out-Relief. Economic Journal 5: 603–623.

Yule, Udny. 1899. An Investigation into the Causes of Changes in Pauperism in England, Chiefly during the Last Tw o Intercensal Decades (Part I). Journal of the Royal Statistical Society 62: 249–295.

William D. Lastrapes