Covariance estimation on matrix manifolds

Abstract

The estimation of covariance matrices is a fundamental problem in multivariate analysis and uncertainty quantification. Covariance matrices are an essential modeling tool in climatology, econometrics, model reduction, biostatistics, signal processing, and geostatistics, among other applications. In practice, covariances often must be estimated from samples. While the sample covariance matrix is a consistent estimator, it performs poorly when the relative number of samples is small; improved estimators that impose structure must be considered. Yet standard parametric covariance families can be insufficiently flexible for many applications, and non-parametric approaches may not easily allow certain kinds of prior knowledge to be incorporated. In this thesis, we harness the structure of the manifold of symmetric positive-(semi)definite matrices to build families of covariance matrices out of geodesic curves.

These covariance families offer more flexibility for problem-specific tailoring than classical parametric families, and are preferable to simple convex combinations. Moreover, the proposed families can be interpretable: the internal parameters may serve as explicative variables for the problem of interest. Once a covariance family has been chosen, one typically needs to select a representative member by solving an optimization problem, e.g., by maximizing the likelihood associated with a data set. Consistent with the construction of the covariance family, we propose a differential geometric interpretation of this problem: minimizing the natural distance on the covariance manifold. Our approach does not require assuming a particular probability distribution for the data. Within this framework, we explore two different estimation settings.

First, we consider problems where representative "anchor" covariance matrices are available; these matrices may result from offline empirical observations or computational simulations of the relevant spatiotemporal process at related conditions. We connect multiple anchors to build multi-parametric covariance families, and then project new observations onto this family--for instance, in online estimation with limited data. We explore this problem in the full-rank and low-rank settings. In the former, we show that the proposed natural distance-minimizing projection and maximum likelihood are locally equivalent up to second order. In the latter, we devise covariance families and minimization schemes based on generalizations of multi-linear and Bézier interpolation to the appropriate manifold.

Second, for problems where anchor matrices are unavailable, we propose a geodesic reformulation of the classical shrinkage estimator: that is, we construct a geodesic family that connects the identity (or any other target) matrix to the sample covariance matrix and minimize the expected natural distance to the true covariance. The proposed estimator inherits the properties of the geodesic distance, for instance, invariance to inversion. Leveraging previous results, we propose a solution heuristic that compares favorably with recent non-linear shrinkage estimators. We demonstrate these covariance families and estimation approaches in a range of synthetic examples, and in applications including wind field modeling and groundwater hydrology.