Image sequence segmentation using multiple features and edge fusion: its algorithm and VLSI architecture

Abstract

Semantic object representation is an important step for digital multimedia applications such as object-based coding, content-based access, and manipulations. This dissertation presents an image sequence segmentation algorithm and its VLSI architecture which provides initial region information for the video coding and the semantic object representation in image sequences. Our objective is to develop a hardware-friendly segmentation algorithm and its architecture by combining static and dynamic features simultaneously in one scheme. In the initial stage of the algorithm, a multiple feature space is transformed to a label space by using the self-organizing feature maps (SOFM) neural networks. The next stage is an edge fusion in which edge information is incorporated into the neural network outputs to generate more precisely located boundaries of segmentation. Pixel-based feature vectors consisting of three color, motion, and two texture features are extracted from two frames of an image sequence. These feature vectors are smoothed and normalized. A soft weighting scheme is applied to the normalized features. The weighting scheme suppresses unreliable feature components in a feature vector by making their values low. In order to generate the segmentation label space, the weighted multiple feature space is transformed to the one-dimensional label space using the SOFM neural networks. The oversegmented segmentation labels are further processed by incorporating edge information in order to generate segmented region boundaries closer to edges. The edge fusion is an iterative region merging process using a similarity criterion consisting of color difference, region geometry, and edge information between two regions. Experimental results for a variety of MPEG image sequences are evaluated and compared with an existing segmentation method to clarify the advantages of the proposed algorithm objectively. The proposed algorithm differs from existing methods as followings: (1) it can segment textured images with low-dimensional texture features, (2) it leads to more meaningful segmentation region boundaries, and (3) it is easier to be mapped into hardware than existing methods. Also, this dissertation proposes a VLSI segmentation architecture of the proposed algorithm. The proposed segmentation scheme is mapped into a dedicated hardware system. The dedicated special-purpose system consists of motion estimation, edge detection, edge linking, median and min filters, feature normalization and weighting, the systolic feature labeling, and edge fusion subsystems which can be easily mapped into systolic and pipelined architectures. Computational and hardware complexities of the proposed system architecture are estimated in terms of the number of clock cycles, arithmetic components and memory requirement. The proposed VLSI architecture makes it possible to perform image sequence segmentation in real-time.