奇异值分解

维基百科，自由的百科全书

奇异值分解是线性代数中一种重要的矩阵分解，在信号处理、统计学等领域有重要应用。奇异值分解在某些方面与对称矩阵或Hermite矩阵基于特征向量的对角化类似。然而这两种矩阵分解尽管有其相关性，但还是有明显的不同。对称阵特征向量分解的基础是谱分析，而奇异值分解则是谱分析理论在任意矩阵上的推广。

目录 1 Statement of the theorem 2 奇异值和奇异向量, 以及他们与奇异值分解的关系 3 Relation to eigenvalue decomposition 4 几何意义 5 Reduced SVDs 5.1 Thin SVD 5.2 Compact SVD 5.3 Truncated SVD 6 Norms 7 应用 7.1 求伪逆 7.2 值域、零空间和秩 7.3 Matrix approximation 7.4 Other examples 8 Computation of the SVD 9 歴史 10 See also 11 External links 12 参考文献

[编辑] Statement of the theorem

Suppose M is an m-by-n matrix whose entries come from the field K, which is either the field of real numbers or the field of complex numbers. Then there exists a factorization of the form

where U is an m-by-m unitary matrix over K, the matrix Σ is m-by-n with nonnegative numbers on the diagonal and zeros off the diagonal, and V^* denotes the conjugate transpose of V, an n-by-n unitary matrix over K. Such a factorization is called a singular-value decomposition of M.

• The matrix V thus contains a set of orthogonal "input" or "analysing" base-vector directions for M
• The matrix U contains a set of orthogonal "output" base-vector directions for M
• The matrix Σ contains the singular values, which can be thought of as scalar "gain controls" by which each corresponding input is multiplied to give a corresponding output.

A common convention is to order the values Σ_i,i in non-increasing fashion. In this case, the diagonal matrix Σ is uniquely determined by M (though the matrices U and V are not).

[编辑] 奇异值和奇异向量, 以及他们与奇异值分解的关系

A non-negative real number σ is a singular value for M if and only if there exist normalized vectors u in K^m and v in Kⁿ such that

The vectors u and v are called left-singular and right-singular vectors for σ, respectively.

In any singular value decomposition

the diagonal entries of Σ are necessarily equal to the singular values of M. The columns of U and V are left- resp. right-singular vectors for the corresponding singular values. Consequently, the above theorem states that

• An m × n matrix M has at least one and at most p = min(m,n) distinct singular values.

• It is always possible to find a unitary basis for K^m consisting of left-singular vectors of M.

• It is always possible to find a unitary basis for Kⁿ consisting of right-singular vectors of M.

A singular value for which we can find two left (or right) singular vectors that are not linearly dependent is called degenerate.

Non-degenerate singular values always have unique left and right singular vectors, up to multiplication by a unit phase factor e^iφ (for the real case up to sign). Consequently, if all singular values of M are non-degenerate and non-zero, then its singular value decomposition is unique, up to multiplication of a column of U by a unit phase factor and simultaneous multiplication of the corresponding column of V by the same unit phase factor.

Degenerate singular values, by definition, have non-unique singular vectors. Furthermore, if u₁ and u₂ are two left-singular vectors which both correspond to the singular value σ, then any normalized linear combination of the two vectors is also a left singular vector corresponding to the singular value σ. The similar statement is true for right singular vectors. Consequently, if M has degenerate singular values, then its singular value decomposition is not unique.

[编辑] Relation to eigenvalue decomposition

The singular value decomposition is very general in the sense that it can be applied to any m × n matrix. The eigenvalue decomposition, on the other hand, can only be applied to certain classes of square matrices. Nevertheless, the two decompositions are related.

In the special case that M is a Hermitian matrix which is positive semi-definite, i.e., all its eigenvalues are real and non-negative, then the singular values and singular vectors coincide with the eigenvalues and eigenvectors of M,

More generally, given an SVD of M, the following two relations hold:

The right hand sides of these relations describe the eigenvalue decompositions of the left hand sides. Consequently, the squares of the non-zero singular values of M are equal to the non-zero eigenvalues of either M*M or MM*. Furthermore, the columns of U (left singular vectors) are eigenvectors of MM* and the columns of V (right singular vectors) are eigenvectors of M*M.

[编辑] 几何意义

因为U 和V 向量都是单位化的向量, 我们知道U的列向量u₁,...,u_m组成了K^m空间的一组标准正交基。同样，V的列向量v₁,...,v_n也组成了Kⁿ空间的一组标准正交基(根据向量空间的标准点积法则).

线性变换T: Kⁿ → K^m，把向量x变换为Mx。考虑到这些标准正交基，这个变换描述起来就很简单了: T(v_i) = σ_i u_i, for i = 1,...,min(m,n), 其中σ_i 是对角阵Σ中的第i个元素; 当i > min(m,n)时，T(v_i) = 0。

这样，SVD理论的几何意义就可以做如下的归纳：对于每一个线性映射T: Kⁿ → K^m，T把Kⁿ的第i个基向量映射为K^m的第i个基向量的非负倍数,然后将余下的基向量映射为零向量。对照这些基向量，映射T就可以表示为一个非负对角阵。

[编辑] Reduced SVDs

In applications it is quite unusual for the full SVD, including a full unitary decomposition of the null-space of the matrix, to be required. Instead, it is often sufficient (as well as faster, and more economical for storage) to compute a reduced version of the SVD. The following can be distinguished for an m×n matrix M of rank r:

[编辑] Thin SVD

Only the n column vectors of U corresponding to the row vectors of V* are calculated. The remaining column vectors of U are not calculated. This is significantly quicker and more economical than the full SVD if n<<m. The matrix U_n is thus m×n, Σ_n is n×n diagonal, and V is n×n.

The first stage in the calculation of a thin SVD will usually be a QR decomposition of M, which can make for a significantly quicker calculation if n<<m.

[编辑] Compact SVD

Only the r column vectors of U and r row vectors of V* corresponding to the non-zero singular values Σ_r are calculated. The remaining vectors of U and V* are not calculated. This is quicker and more economical than the thin SVD if r<<n. The matrix U_r is thus m×r, Σ_r is r×r diagonal, and V_r* is r×n.

[编辑] Truncated SVD

Only the t column vectors of U and t row vectors of V* corresponding to the t largest singular values Σ_r are calculated. The rest of the matrix is discarded. This can be much quicker and more economical than the thin SVD if t<<r. The matrix U_t is thus m×t, Σ_t is t×t diagonal, and V_t* is t×n'.

Of course the truncated SVD is no longer an exact decomposition of the original matrix M, but as discussed below, the approximate matrix is in a very useful sense the closest approximation to M that can be achieved by a matrix of rank t.

[编辑] Norms

The sum of the k largest singular values of M is a matrix norm, the Ky Fan k-norm of M. The Ky Fan 1-norm is just the operator norm of M as a linear operator with respect to the Euclidean norms of K^m and Kⁿ. The square root of the sum of squares of the singular values is the Frobenius norm of M.

[编辑] 应用

[编辑] 求伪逆

奇异值分解可以被用来计算矩阵的伪逆。若矩阵 M 的奇异值分解为 M = UΣV ^* ，那么 M 的伪逆为

其中 Σ⁺ 是将Σ转置，并将其主对角线上每个非零元素都求倒数得到的。求伪逆通常可以用来求解线性最小平方问题。

[编辑] 值域、零空间和秩

Another application of the SVD is that it provides an explicit representation of the range and null space of a matrix M. The right singular vectors corresponding to vanishing singular values of M span the null space of M. The left singular vectors corresponding to the non-zero singluar vectors of M span the range of M. As a consequence, the rank of M equals the number of non-zero singular values of M. In numerical linear algebra the singular values can be used to determine the effective rank of a matrix, as rounding error may lead to small but non-zero singular values in a rank deficient matrix.

[编辑] Matrix approximation

Some practical applications need to solve the problem of approximating a matrix M with another matrix which has a specific rank r. In the case that the approximation is based on minimizing the Frobenius norm of the difference between M and under the constraint that it turns out that the solution is given by the SVD of M, namely

where is the same matrix as Σ except that it contains only the r largest singular values (the other singular values are replaced by zero).

[编辑] Other examples

The SVD is also applied extensively to the study of linear inverse problems, and is useful in the analysis of regularization methods such as that of Tikhonov. It is widely used in statistics where it is related to principal component analysis, and in signal processing and pattern recognition. It is also used in output-only modal analysis, where the non-scaled mode shapes can be determined from the singular vectors.

One application of SVD to rather large matrices is in numerical weather prediction, where Lanczos methods are used to estimate the most linearly quickly growing few perturbations to the central numerical weather prediction over a given initial forward time period -- ie the singular vectors corresponding to the largest singular values of the linearised propagator for the global weather over that time interval. The output singular vectors in this case are entire weather systems! These perturbations are then run through the full nonlinear model to generate an ensemble forecast, giving a handle on some of the uncertainty that should be allowed for around the current central prediction.

[编辑] Computation of the SVD

The LAPACK subroutine DGESVD represents a typical approach to the computation of the singular value decomposition. If the matrix has more rows than columns a QR decomposition is first performed. The factor R is then reduced to a bidiagonal matrix. The desired singular values and vectors are then found by performing a bidiagonal QR iteration, using the LAPACK routine DBDSQR (See Demmel and Kahan for details).

The GNU Scientific Library offers four functions, one with the Golub-Rinsch algorithm, one with the modified Golub-Rinsch algorithm (faster for matrix with much larger height than width), one with a one-sided Jacobi orthogonalization and one which uses only non-zero singular values. See the GSL manual page on SVD.

[编辑] 歴史

The singular value decomposition was originally developed by differential geometers, who wished to determine whether a real bilinear forms could be made equal to another by independent orthogonal transformations of the two spaces it acts on. Eugenio Beltrami and Camille Jordan discovered independently, in 1873 and 1874 respectively, that the singular values of the bilinear forms, represented as a matrix, form a complete set of invariants for bilinear forms under orthogonal substitutions. James Joseph Sylvester also arrived at the singular value decomposition for real square matrices in 1889, apparently independent of both Beltrami and Jordan. Sylvester called the singular values the canonical multipliers of the matrix A. The fourth mathematician to discover the singular value decomposition independently is Autonne in 1915, who arrived at it via the polar decomposition. The first proof of the singular value decomposition for rectangular and complex matrices seems to be by Eckart and Young in 1939; they saw it as a generalization of the principal axis transformation for Hermitian matrices.

In 1907, Erhard Schmidt defined an analog of singular values for integral operators; it seems he was unaware of the parallel work on singular values of finite matrices. This theory was further developed by Émile Picard in 1910, who is the first to call the numbers σ_k singular values (or rather, valeurs singulières).

[编辑] See also

• matrix decomposition
• s-number
• generalized singular value decomposition
• polar decomposition
• empirical orthogonal functions (EOFs)
• canonical correlation analysis (CCA)
• latent semantic analysis

[编辑] External links

• LAPACK users manual gives details of subroutines to calculate the SVD (see also [1]).
• Applications of SVD on PC Hansen's web site.
• Introduction to the Singular Value Decomposition by Todd Will of the University of Wisconsin--La Crosse.
• Los Alamos group's book chapter has helpful gene data analysis examples.
• MIT Lecture series by Gilbert Strang. See Lecture #29 on the SVD.
• Java SVD library routine.
• Java applet demonstrating the SVD.
• Java script demonstrating the SVD more extensively, paste your data from a spreadsheet.
• Chapter from "Numerical Recipes in C" gives more information about implementation and applications of SVD.
• Online Matrix Calculator Performs singular value decomposition of matrices.

[编辑] 参考文献

• Demmel, J. and Kahan, W. (1990). Computing Small Singular Values of Bidiagonal Matrices With Guaranteed High Relative Accuracy. SIAM J. Sci. Statist. Comput., 11 (5), 873-912.
• Golub, G. H. and Van Loan, C. F. (1996). "Matrix Computations". 3rd ed., Johns Hopkins University Press, Baltimore. ISBN 0801854148.
• Halldor, Bjornsson and Venegas, Silvia A. (1997). "A manual for EOF and SVD analyses of climate data". McGill University, CCGCR Report No. 97-1, Montréal, Québec, 52pp.
• Hansen, P. C. (1987). The truncated SVD as a method for regularization. BIT, 27, 534-553.
• Horn, Roger A. and Johnson, Charles R (1985). "Matrix Analysis". Section 7.3. Cambridge University Press. ISBN 0-521-38632-2.
• Horn, Roger A. and Johnson, Charles R (1991). Topics in Matrix Analysis, Chapter 3. Cambridge University Press. ISBN 0-521-46713-6.
• Strang G (1998). "Introduction to Linear Algebra". Section 6.7. 3rd ed., Wellesley-Cambridge Press. ISBN 0961408855.