Covariances in Computer Vision and Machine Learning

Covariance matrices play important roles in many areas of mathematics, statistics, and machine learning, as well as their applications. In computer vision and image processing, they give rise to a powerful data representation, namely the covariance descriptor, with numerous practical applications.

In this book, we begin by presenting an overview of the {\it finite-dimensional covariance matrix} representation approach of images, along with its statistical interpretation. In particular, we discuss the various distances and divergences that arise from the intrinsic geometrical structures of the set of Symmetric Positive Definite (SPD) matrices, namely Riemannian manifold and convex cone structures. Computationally, we focus on kernel methods on covariance matrices, especially using the Log-Euclidean distance.

We then show some of the latest developments in the generalization of the finite-dimensional covariance matrix representation to the {\it infinite-dimensional covariance operator} representation via positive definite kernels. We present the generalization of the affine-invariant Riemannian metric and the Log-Hilbert-Schmidt metric, which generalizes the Log-Euclidean distance. Computationally, we focus on kernel methods on covariance operators, especially using the Log-Hilbert-Schmidt distance. Specifically, we present a two-layer kernel machine, using the Log-Hilbert-Schmidt distance and its finite-dimensional approximation, which reduces the computational complexity of the exact formulation while largely preserving its capability. Theoretical analysis shows that, mathematically, the approximate Log-Hilbert-Schmidt distance should be preferred over the approximate Log-Hilbert-Schmidt inner product and, computationally, it should be preferred over the approximate affine-invariant Riemannian distance.

Numerical experiments on image classification demonstrate significant improvements of the infinite-dimensional formulation over the finite-dimensional counterpart. Given the numerous applications of covariance matrices in many areas of mathematics, statistics, and machine learning, just to name a few, we expect that the infinite-dimensional covariance operator formulation presented here will have many more applications beyond those in computer vision.

Ha Quang Minh received the Ph.D. degree in mathematics from Brown University, Providence, RI, USA, in May 2006, under the supervision of Steve Smale. He is currently a Researcher in the Department of Pattern Analysis and Computer Vision (PAVIS) with the Istituto Italiano di Tecnologia (IIT), Genova, Italy. Prior to joining IIT, he held research positions at the University of Chicago, the University of Vienna, Austria, and Humboldt University of Berlin, Germany. He was also a Junior Research Fellow at the Erwin Schrodinger International Institute for Mathematical Physics in Vienna and a Fellow at the Institute for Pure and Applied Mathematics (IPAM) at the University of California, Los Angeles (UCLA). His current research interests include applied and computational functional analysis, applied and computational di erential geometry, machine learning, computer vision, and image and signal processing. His recent research contributions include the infinite-dimensional Log-Hilbert-Schmidt metric and Log-Determinant divergences between positive definite operators, along with their applications in machine learning and computer vision in the setting of kernel methods. He received the Microsoft Best Paper Award at the Conference on Uncertainty in Artificial Intelligence (UAI) in 2013 and the IBM Pat Goldberg Memorial Best Paper Award in Computer Science, Electrical Engineering, and Mathematics in 2013.