Generalized least squares

This article may require cleanup to meet Wikipedia's quality standards. No cleanup reason has been specified. Please help improve this article if you can. (May 2010) (Learn how and when to remove this message)

This article needs additional citations for verification. Please help improve this article by adding citations to reliable sources. Unsourced material may be challenged and removed. Find sources: "Generalized least squares" – news · newspapers · books · scholar · JSTOR (July 2009) (Learn how and when to remove this message)

In statistics, generalized least squares (GLS) is a technique for estimating the unknown parameters in a linear regression model. The GLS is applied when the variances of the observations are unequal (heteroscedasticity), or when there is a certain degree of correlation between the observations. In these cases ordinary least squares can be statistically inefficient, or even give misleading inferences.

In a typical linear regression model we observe data ${\displaystyle \{y_{i},x_{ij}\}_{i=1..n,j=1..p}}$ on n statistical units. The response values are placed in a vector Y = (y₁, ..., y_n)′, and the predictor values are placed in the design matrix X = [[x_ij]], where x_ij is the value of the jth predictor variable for the ith unit. The model assumes that the conditional mean of Y given X is a linear function of X, whereas the conditional variance of Y given X is a known matrix Ω. This is usually written as

${\displaystyle Y=X\beta +\varepsilon ,\qquad \mathrm {E} [\varepsilon |X]=0,\ \operatorname {Var} [\varepsilon |X]=\Omega .}$

Here β is a vector of unknown “regression coefficients” that must be estimated from the data.

Suppose b is a candidate estimate for β. Then the residual vector for b will be Y − Xb. Generalized least squares method estimates β by minimizing the squared Mahalanobis length of this residual vector:

${\displaystyle {\hat {\beta }}={\underset {b}{\rm {arg\,min}}}\,(Y-Xb)'\,\Omega ^{-1}(Y-Xb),}$

Since the objective is a quadratic form in b, the estimator has an explicit formula:

${\displaystyle {\hat {\beta }}=(X'\Omega ^{-1}X)^{-1}X'\Omega ^{-1}Y.}$

The GLS estimator is unbiased, consistent, efficient, and asymptotically normal:

${\displaystyle {\sqrt {n}}({\hat {\beta }}-\beta )\ {\xrightarrow {d}}\ {\mathcal {N}}\!\left(0,\,(X'\,\Omega ^{-1}X)^{-1}\right).}$

GLS is equivalent to applying ordinary least squares to a linearly transformed version of the data. To see this, factor Ω = BB′, for instance using the Cholesky decomposition. Then if we multiply both sides of the equation Y = Xβ + ε by B⁻¹, we get an equivalent linear model Y* = X*β + ε*, where Y* = B⁻¹Y, X* = B⁻¹X, and ε* = B⁻¹ε. In this model Var[ε*] = B⁻¹ΩB⁻¹ = I. Thus we can efficiently estimate β by applying OLS to the transformed data, which requires minimizing

${\displaystyle (Y^{*}-X^{*}b)'(Y^{*}-X^{*}b)=(Y-Xb)'\,\Omega ^{-1}(Y-Xb).}$

This has the effect of standardizing the scale of the errors and “de-correlating” them. Since OLS is applied to data with homoscedastic errors, the Gauss–Markov theorem applies, and therefore the GLS estimate is the best linear unbiased estimator for β.

A special case of GLS called weighted least squares occurs when all the off-diagonal entries of Ω are 0. This situation arises when the variances of the observed values are unequal (i.e. heteroscedasticity is present), but where no correlations exist among the observed values. The weight for unit i is proportional to the reciprocal of the variance of the response for unit i.

This article needs attention from an expert on the subject. Please add a reason or a talk parameter to this template to explain the issue with the article. When placing this tag, consider associating this request with a WikiProject. (May 2010)

Feasible generalized least squares is similar to generalized least squares except that it uses an estimated variance-covariance matrix since the true matrix is not known directly.

The ordinary least squares (OLS) estimator is calculated as usual by

${\displaystyle {\widehat {\beta }}_{OLS}=(X'X)^{-1}X'y}$

and estimates of the residuals ${\displaystyle {\widehat {u}}_{j}=(Y-Xb)_{j}}$ are constructed.

Construct ${\displaystyle {\widehat {\Omega }}_{OLS}}$:

${\displaystyle {\widehat {\Omega }}_{OLS}=\operatorname {diag} ({\widehat {u}}_{1}^{2},{\widehat {u}}_{2}^{2},\dots ,{\widehat {u}}_{n}^{2}).}$

Estimate ${\displaystyle \beta _{FGLS1}}$ using ${\displaystyle {\widehat {\Omega }}_{OLS}}$ using weighted least squares

${\displaystyle {\widehat {\beta }}_{FGLS1}=(X'{\widehat {\Omega }}_{OLS}^{-1}X)^{-1}X'{\widehat {\Omega }}_{OLS}^{-1}y}$

${\displaystyle {\widehat {u}}_{FGLS1}=Y-X{\widehat {\beta }}_{FGLS1}}$

${\displaystyle {\widehat {\Omega }}_{FGLS1}=\operatorname {diag} ({\widehat {u}}_{FGLS1,1}^{2},{\widehat {u}}_{FGLS1,2}^{2},\dots ,{\widehat {u}}_{FGLS1,n}^{2})}$

${\displaystyle {\widehat {\beta }}_{FGLS2}=(X'{\widehat {\Omega }}_{FGLS1}^{-1}X)^{-1}X'{\widehat {\Omega }}_{FGLS1}^{-1}y}$

This estimation of ${\displaystyle {\widehat {\Omega }}}$ can be iterated to convergence given that the assumptions outlined in White^{[citation needed]} hold.

The WLS and FGLS estimators have the following distributions

${\displaystyle {\widehat {\beta }}_{WLS}~\sim N(\beta ,(X'\Omega ^{-1}X)^{-1})}$

${\displaystyle {\widehat {\beta }}_{FGLS}~\sim N(\beta ,(X'{\widehat {\Omega }}_{OLS}^{-1}X)^{-1}(X'{\widehat {\Omega }}_{OLS}^{-1}\Omega {\widehat {\Omega }}_{OLS}^{-1}X)(X'{\widehat {\Omega }}_{OLS}^{-1}X)^{-1})}$

• Iteratively reweighted least squares
• Confidence region
• Effective degrees of freedom