Robust and Efficient Multi-target Video Tracking Based on Multi-feature Fusion and State Estimation Fusion

Abstract

Video tracking occupies an extremely important position in computer vision, and it has been widely applied to military and civil fields. However, video tracking needs a large number of calculations due to complex image processing and computer vision algorithms. In addition, video tracking needs to face various complex scenarios which pose great challenges to the robustness of tracking algorithms. In this thesis, an efficient and robust multi-target video detection and tracking framework, which integrates automatic video target detection, multi-feature fusion based video target modelling, multi-target data association, video target management, state estimation fusion, and distributed multi-camera tracking, is presented. Firstly, an automatic, robust, and efficient target detection approach is proposed. The Canny edge detector and the simplified multi-scale wavelet decomposition are exploited to simultaneously extract the contour of targets. Also, efficient background modelling based on improved Gaussian mixture models (IGMMs) is investigated to implement background subtraction (BGS) and to segment the foreground. Compared with traditional GMM, IGMMs improves the initialization process and optimizes the background-pixel matching strategy by using the mesh-updating technique. In addition, three-consecutive-frame difference (TCFD) is integrated with the proposed IGMMs-based BGS to quickly locate video targets. Moreover, fast morphological operations are performed on monochrome foreground images to segment targets-of-interest and to extract corresponding contours. After that, multi-feature fusion-based target modelling is introduced to robustly describe video targets. The spatial colour distribution, rotation-and-scale invariant as well as uniform local binary pattern (RSIULBP) texture, and edge orientation gradients are calculated and fused to build a fused-feature matching matrix which is integrated into data associations to realize reliable and precise multi-target tracking. In addition, low-dimensional regional covariance matrices-based multi-feature fusion is exploited to improve the matching degree of targets in single target tracking. Parallel computing based on multi-threaded synchronization is employed to boost the efficiency of feature extraction and fusion. An accurate and efficient multi-target data association method that integrates an improved probabilistic data association (IPDA) and a simplified joint probabilistic data association (SJPDA) is designed in this study. IPDA combines the augmented posterior probability matrix with the fused-feature matching matrix to perform multi-target associations. SJPDA ensures the efficiency of data associations and yields a better accuracy in the presence of low PSNR and sparse targets by sifting out big probability events. In order to record and update target trajectories, as well as increase the accuracy of multi-target tracking, a video target management scheme is presented. The states throughout the whole lifecycle of targets are defined and analysed. Meanwhile, a prediction interpolation-based data recovery approach is discussed to restore missed measurements. Afterwards, a flexible and extensible data structure is designed to encapsulate target states at each time step. Variable-length sequence containers are exploited to store existing targets, newly appearing targets, and targets which have disappeared. The switching criterion of target states is discussed. To quickly and robustly estimate the motion states of rigid targets, mixed Kalman/ H∞ filtering based on state covariances fusion and state estimates fusion is proposed. The H∞ filter makes no assumptions about process and measurement noise, and it has similar recursive equations to the Kalman filter. Thus, it is more robust against non-Gaussian noise. The mixed Kalman/H∞ filter can guarantee both the efficiency and robustness of state estimations under uncertain noise. To predict the state of high-maneuvering targets, mixed extended Kalman/particle filtering is introduced. The extended Kalman filter is able to linearize system dynamic models using Taylor series expansion. Hence it can implement a slightly nonlinear state estimation. An improved sequential importance resampling particle filtering is discussed to estimate target states in the case of strong nonlinearity and dynamic background. The mixed extended Kalman/particle filtering is performed by feeding the state output of the extended Kalman filter back to the particle filter to initialize the deployment of particles. Compared with single-camera video tracking, multi-camera tracking retrieves more information about the targets-of-interest from different perspectives and can better solve the problem of target occlusions. A multi-camera cooperative tracking strategy is investigated and a relay tracking scheme based on improved Camshift is proposed. To further extend the scope of tracking, a distributed multi-camera video tracking and surveillance (DMVTS) system based on hierarchical centre management modules is developed.