Data Association for Multi-object Visual Tracking

In the human quest for scientific knowledge, empirical evidence is collected by visual perception. Tracking with computer vision takes on the important role to reveal complex patterns of motion that exist in the world we live in. Multi-object tracking algorithms provide new information on how groups and individual group members move through three-dimensional space. They enable us to study in depth the relationships between individuals in moving groups. These may be interactions of pedestrians on a crowded sidewalk, living cells under a microscope, or bats emerging in large numbers from a cave. Being able to track pedestrians is important for urban planning; analysis of cell interactions supports research on biomaterial design; and the study of bat and bird flight can guide the engineering of aircraft. We were inspired by this multitude of applications to consider the crucial component needed to advance a single-object tracking system to a multi-object tracking system―data association.

Data association in the most general sense is the process of matching information about newly observed objects with information that was previously observed about them. This information may be about their identities, positions, or trajectories. Algorithms for data association search for matches that optimize certain match criteria and are subject to physical conditions. They can therefore be formulated as solving a "constrained optimization problem"―the problem of optimizing an objective function of some variables in the presence of constraints on these variables. As such, data association methods have a strong mathematical grounding and are valuable general tools for computer vision researchers.

This book serves as a tutorial on data association methods, intended for both students and experts in computer vision. We describe the basic research problems, review the current state of the art, and present some recently developed approaches. The book covers multi-object tracking in two and three dimensions. We consider two imaging scenarios involving either single cameras or multiple cameras with overlapping fields of view, and requiring across-time and across-view data association methods. In addition to methods that match new measurements to already established tracks, we describe methods that match trajectory segments, also called tracklets. The book presents a principled application of data association to solve two interesting tasks: first, analyzing the movements of groups of free-flying animals and second, reconstructing the movements of groups of pedestrians. We conclude by discussing exciting directions for future research.

Margrit Betke is a Professor of Computer Science at Boston University, where she co-leads the Image and Video Computing Research Group. She earned her Ph.D. degree in Computer Science and Electrical Engineering at the Massachusetts Institute of Technology in 1995. She conducts research in computer vision, in particular, the development of methods for detection, segmentation, registration, and tracking of objects invisible-light, infrared, and x-ray image data. She has worked on gesture, vehicle, and animal tracking, video-based human-computer interfaces, statistical object recognition, and medical imaging analysis. Prof. Betke has co-invented the ""Camera Mouse,"" an assistive technology used worldwide by children and adults with severe motion impairments. She co-developed the first patented algorithms for detecting and measuring pulmonary nodule growth in computed tomography. She recently led a six-year research program to develop intelligent tracking systems that reason about group behavior of people, bats, birds, and cells. She has published over 140 original research papers.Zheng Wu is a Senior Computer Vision Engineer at The Mathworks, Inc., in Natick, Massachusetts, which is the producer of the widely used computing environment MATLAB. Before joining Mathworks, Dr. Wu was a postdoctoral associate in the Department of Computer Science at Boston University, where he earned his Ph.D. degree in 2012. He was a member of the Image and Video Computing research group at Boston University, working with Profs. Margrit Betke and Stan Sclaroff and their team of students. Dr. Wus general research interests include computer vision, machine learning, and combinatorial algorithms. In his work, he has focused on object segmentation, detection, and tracking, and has published over 20 original papers on these topics.