Variance-Covariance Matrix
Variance-Covariance Matrix
ADDITIONAL MATHEMATICAL PROPERTIES
The variance-covariance matrix is a convenient expression of statistics in data describing patterns of variability and covariation. The variance-covariance matrix is widely used both as a summary statistic of data and as the basis for key concepts in many multivariate statistical models.
VERBAL DEFINITION
The variance-covariance matrix, often referred to as Cov(), is an average cross-products matrix of the columns of a data matrix in deviation score form. A deviation score matrix is a rectangular arrangement of data from a study in which the column average taken across rows is zero. The variance-covariance matrix expresses patterns of variability as well as covariation across the columns of the data matrix. In most contexts the (vertical) columns of the data matrix consist of variables under consideration in a study and the (horizontal) rows represent individual records. Variance-covariance matrices may, however, be calculated from any pairwise combination of individuals, measurement occasions, or variables. Even this by no means exhausts the possible covariance matrices that may be considered for a statistical model (see Cattell [1988] for an extensive list of the possibilities involving several dimensions).
MATHEMATICAL DEFINITION
If k variables are assumed in a study and letting X denote a raw score version of the data matrix and µk a vector of means for the variables under consideration, the covariance matrix is defined as E (X′X ) – µk ′µk, where E () denotes the expectation operator. If the columns of X are centered to a mean of 0, the variance-covariance matrix is more conveniently expressed as E (x′x ). Within linear algebra, covariance matrices belong to the class of matrices known as nonnegative-definite symmetric matrices.
CALCULATION
The sample covariance matrix can be calculated as 1/n *1k *(X′X – µk ′µk ) if raw score matrix X is used, µk denotes a vector of sample means, and where ’ denotes the transpose operator. If data are expressed in column-centered form, x = (X – X ), Cov(x ) is calculated as 1/n *1k *x’x, where 1k denotes a k -column vector of 1s and n denotes the number of observations. The sample variance-covariance matrix, although efficient, is a biased estimate of population variability. As a result, the estimated population covariance matrix divides by the reciprocal of n – 1 of n. If the x matrix is further transformed to have a variance of 1 (usually termed Zx ), the resulting sample Cov() matrix is known as a correlation matrix. If the X matrix is retained in raw score form and an additional unit column is added to the data, 1/n *1k *X′X is referred to as an average sum of squares and cross-products matrix, a data summary convenient for models in which overall elevation as well as patterns of covariation are of interest, as often occurs in longitudinal studies of growth.
PROPERTIES
The Cov() matrix has as many rows and columns as the columns of X and is symmetric (meaning that the value associated with the j th row and k th column in Cov() is equal to the value in the k th row and j th column). Diagonal elements of Cov() represent the variances of the column variables; off-diagonal elements represent covariances or, if based on Zx, correlation coefficients.
ADDITIONAL MATHEMATICAL PROPERTIES
Covariance of a sum: Assuming three matrices x 1, x 2, and y,
Cov(x 1 + x 2, y ) = Cov(x 1, y ) + Cov(x 2, y ).
Covariances involving matrix products: Assuming two conformable matrices A and B,
Cov (AX,BX ) = A Cov(X,X )B’ where ’ denotes the transpose operator.
USES
As mentioned before, covariance matrices, by themselves, are compact summaries of the variability and covariation present in data. More generally, the covariance matrix and vector of means constitute sufficient statistics for models that assume a multivariate normal distribution. As such, the covariance matrix may be used in lieu of the raw data in calculating a number of multivariate statistical models, such as confirmatory and exploratory factor analysis (assuming the diagonal of the matrix is appropriately adjusted by the estimated communality), path analysis, or other general linear models, including the special cases of multiple regression, analysis of variance, and repeated measures analysis of variance or MANOVA. In many statistical models, finding an optimal basis for representing the covariance matrix in a compact fashion is of primary interest. Such reduced or optimal bases are referred to as principal components analysis (PCA), or in image processing as the Karhunen-Loève transform, which has time series applications within psychology (Molenaar and Boomsma 1987). Covariance matrices alone are not sufficient statistics for other more sophisticated models, such as those involving weighted least squares, sampling weights, or other categorical or distributional adjustments to reflect the dichotomous, polytomous, or other distributional characteristics of the variables under consideration.
SEE ALSO Classical Statistical Analysis; Covariance; Econometric Decomposition; Inverse Matrix; Least Squares, Ordinary; Matrix Algebra; Ordinary Least Squares Regression; Path Analysis; Regression; Regression Analysis; Statistics
BIBLIOGRAPHY
Cattell, Raymond B. 1988. The Meaning and Strategic Use of Factor Analysis. In Handbook of Multivariate Experimental Psychology, eds. John R. Nesselroade and Raymond B. Cattell, 174–243. 2nd ed. New York: Plenum.
Molenaar, Peter C. M., and Dorrett I. Boomsma. 1987. The Genetic Analysis of Repeated Measures. II: The Karhunen-Loève Expansion. Behavior Genetics 17 (3): 229–242.
Phillip K. Wood