Nonlinear Regression

BIBLIOGRAPHY

A brief discussion of linear regression is essential in understanding nonlinear regression. One of the assumptions of the classical linear regression model is linearity of the functional form. A linear regression model can be written as:

Y = β ₁X ₁ + β ₂X ₂ + … + β_k X_k + ε,

where Y is the dependent variable, X ₁, X ₂, …, X_k are the explanatory variables, and ε is the error term. A popular statistical technique to estimate the value of the parameters of this model is the classical linear regression where the optimization algorithm applies the least squares errors method to find the best fit. Given the classical assumptions, according to the Gauss-Markov theorem, the least squares coefficients are the best linear unbiased estimator (BLUE) of the population of regression coefficients.

Some regression models are intrinsically nonlinear in their parameters; therefore, application of linear regression estimates generates biased results. Nonlinear regression is an extension of the linear least squares regression for a much larger and general class of functions where the relationship between dependent and independent variable is not linear. As a result, the first-order conditions for least squares estimation of the parameters are nonlinear functions of the parameters. A general form of nonlinear regression equation is:

Y_i = f (x _i , θ ) + ε_i,

where x _i is a (N × 1) vector of independent variables, θ is a (K × 1) parameter vector, and ε _i is an additive error term. A common example of intrinsically nonlinear functions is the Cobb-Douglas production function:

where Q_i is the output of firm i, L_i is the labor input, K_i is the capital input, and ε _i is a multiplicative error term. Parameters of β₁ and β₂ are the elasticities of output with respect to labor and capital. Another example is the consumption function:

which reflects the uncertainty with respect to the way income affects consumption expenditures. In these examples, the terms in the models cannot be rearranged to apply the linear least squares.

Pioneers such as Jennrich (1969) and Malinvaud (1970) have advanced the econometric theory for nonlinear statistical models, while development of computing technology in the last few decades has allowed application of nonlinear models to statistical analysis of complicated relationships between variables. When the nonlinear functions cannot be transformed into linear form, econometricians use two common methods, the nonlinear least squares or the maximum likelihood, to estimate the parameters of the model. These approaches essentially search for a solution to the nonlinear optimization problem by minimizing the sum of squared errors or maximizing the likelihood function. Although there are very few limitations on the functional form of nonlinear regression, the parameters of the model are conceptually estimated in the same way as the linear least squares.

However, reaching a solution to a nonlinear optimization problem is a difficult task. A number of computational methods are available. The Gauss-Newton algorithm, a special case of the more general Newton-Raphson algorithm, is a popular method that linearizes the regression model by approximating F (θ) from a Taylor series expansion around an initial θ to minimize the residual sum of squares function. In the iterative linearization process, the nonlinear least squares method typically begins with guessed starting values for the parameters and computes the corresponding sum of squared residuals to reach a global minimum. Although the Gauss-Newton algorithm is more efficient in reaching a global minimum, it is less likely than the Newton-Raphson algorithm to locate the global minimum instead of a local one. Of course, since nonlinearity of the functional form violates the classical assumptions, the Gauss-Markov theorem does not apply to the nonlinear regression.

Econometricians typically ignore the possibility that the new error term created in the process of linearization may not meet the classical linear regression model assumption. Also, the nonlinear least squares method can become computationally expensive when the number of parameters to be estimated exceeds two. Since desirable properties of classical linear regression do not necessarily remain in the nonlinear least square estimator, the maximum likelihood estimator is often preferred. In fact, the two techniques are computationally similar.

In selecting functional form, several factors must be considered, including theoretical consistency, applicability, flexibility, computational difficulty, and factual conformity (Lau, 1986). While the underlying theoretical relationship between variables is important, various statistical techniques are used to allow the data to determine the appropriate functional form. In spite of its problem with the log of zero and negative values, the most popular technique for testing nonlinearity is the Box-Cox transformation method, which tests restrictions on a more general functional form. Some nonlinear equations, however, can be transformed into classical linear form, which would facilitate the estimation of their parameters through the classical least squares procedure. For example, taking the natural log of the Cobb-Douglas production function results in the following function, which is linear in parameters:

ln Q_i = α + ln β ₁L_i + ln β ₂K_i + ln ε_i.

In this case, the least squares estimates of the transformed variables would have the traditional desirable properties.

Nonlinear regression models are sometimes more consistent with the true relationship between variables and have found many applications in models such as the random parameter, continuous regime switching, and time-varying random parameter. Also, in spite of an over-identification tendency in the nonlinear models, the General Method of Moments (GMM) provides a mechanism to arrive at consistent parameter estimates without the normality assumption requirement. However, the use of nonlinear regression models has a few disadvantages. First, estimation procedures for nonlinear models are more complicated because of minimization of sum of square errors, especially when the number of parameters is large, because of minimization of sum of square errors and difficulty of finding the starting values for the iterative linearization process. Second, some measures of goodness of fit, such as the t-statistic and the F-statistic, are not directly compatible and cannot be used or require modification. Compatibility of other measures of goodness of fit such as the R² is debatable. Third, there is more ambiguity in the interpretation of the coefficients in nonlinear models because the derivatives of the regression are usually a function of θ rather than being constant. Overall, although the parsimony rule suggests that computationally more demanding estimators do not have better statistical properties, nonlinear maximum likelihood estimators tend to have better finite sample properties than simple alternatives.

SEE ALSO Econometric Decomposition; Linear Regression; Linear Systems; Nonlinear Systems; Regression; Regression Analysis; Statistics

BIBLIOGRAPHY

Amemiya, T. 1991. Nonlinear Regression Models. In Handbook of Econometrics, ed. Zvi Griliches and Michael Intriligator, 1: 333–389. Amsterdam: North-Holland.

Davidson, Russell, and James MacKinnon. 1993. Estimation and Inference in Econometrics. Oxford: Oxford University Press.

Gallant, Ronald A. 1987. Nonlinear Statistical Models. New York: Wiley.

Granger, Clive W. J. 1993. Strategies for Modeling Nonlinear Time Series Relationship. Economic Record 69: 233–238.

Greene, William H. 2003. Econometrics Analysis, 5th ed. Upper Saddle River, NJ: Prentice Hall.

Jennrich, Robert I. 1969. Asymptotic Properties of Nonlinear Least Squares Estimators. Annals of Mathematical Statistics 40 (2): 633–643.

Judge, George G., R. Carter Hill, William E. Griffiths, et al. 1985. Introduction to the Theory and Practice of Econometrics, 2nd ed. New York: Wiley.

Lau, Lawrence J. 1986. Functional Forms in Econometric Model Building. In Handbook of Econometrics, eds. Zvi Griliches and Michael Intriligator, 3: 1514–1566. Amsterdam: North-Holland.

Malinvaud, Edmond. 1970. The Consistency of Nonlinear Regressions. Annals of Mathematical Statistics 41: 956–969.

Quandt, Richard E. 1983. Computational Problems and Methods. In Handbook of Econometrics, eds. Zve Griliches and Michael Intriligator, 1: 699–764. Amsterdam: North-Holland.

Akbar Marvasti