Blind Super Resolution of Real-Life Video Sequences

ABSTRACT:

Super resolution (SR) for real-life video sequences is a challenging problem due to complex nature of the motion fields. In this paper, a novel blind SR method is proposed to improve the spatial resolution of video sequences, while the overall point spread function of the imaging system, motion fields, and noise statistics are unknown. To estimate the blur(s), first, a nonuniform interpolation SR method is utilized to upsample the frames, and then, the blur(s) is (are) estimated through a multiscale process. The blur estimation process is initially performed on a few emphasized edges and gradually on more edges as the iterations continue. Also for faster convergence, the blur is estimated in the filter domain rather than the pixel domain. The high-resolution frames are estimated using a cost function that has the fidelity and regularization terms of type Huber–Markov random field to preserve edges and fine details. The fidelity term is adaptively weighted at each iteration using a masking operation to suppress artifacts due to inaccurate motions. Very promising results are obtained for real-life videos containing detailed structures, complex motions, fast-moving objects, deformable regions, or severe brightness changes. The proposed method outperforms the state of the art in all performed experiments through both subjective and objective evaluations.

PROJECT OUTPUT VIDEO:

EXISTING SYSTEM:

In existing system most works on the image and video SR are non-blind, i.e. they do not consider blur identification during the SR reconstruction. These methods assume that the PSFs are either known a priori or negligible, both of which are simplistic assumptions for realistic applications. For images, there has been a significant amount of publications on blind deconvolution and a few on blind SR, However, to the best of our knowledge, there are only two independent works on blind SR for videos, which are discussed next. processing videos with arbitrary local motions rather than images with global and translational motion differences, discussions on YCbCr/RGB color spaces and sequential/central motion estimations, adding a final non-blind frame reconstruction after blur estimation, removing structures finer than the blur support during motion estimation, performance, and using a masking operation during frame reconstruction to suppress artifacts.

DISADVANTAGES OF EXISTING SYSTEM:

In the existing system since motion is estimated globally, all regions having local motions need to be masked out. Therefore, the reconstructed video would be of low quality for all locally-moving objects in the scene which could be the most prominent regions to reconstruct.

PROPOSED SYSTEM:

In our proposed system the input video is first upsampled (in case of SR) using a non-uniform interpolation (NUI) SR method, then an iterative procedure is applied using the following considerations: during the initial iterations, the blur is estimated exclusively using a few emphasized edges while weak structures are smoothed out, the number of contributing edges gradually increases as iterations proceed, structures finer than the blur support are omitted from estimation, the estimation is done in the filter domain rather than pixel domain, and finally the estimation is performed at multiple scales to avoid getting trapped in local minima. The cost function used for frame debluring during the blur estimation process has fidelity and regularization terms both of type Huber-Markov Random Field (HMRF). A fidelity term of this type diminishes outliers caused by inaccurate motion estimation and preserve edges.

ADVANTAGES OF PROPOSED SYSTEM:

In this paper, we assume that either the motion blur is global, or the camera’s exposure time is high enough

MODULES:

• Video process

• Upsample/Interpolation

• Edge preserving and smoothing

• HMRF

• Deconvolution

• Super resolution

MODULES DESCSRIPTION:

Video process:

Video processing is a particular case of signal processing, which often employs video filters and where the input and output signals are video files or video streams. Video processing techniques are used in television sets, VCRs, DVDs, video codecs, video players, video scalers and other devices. For example—commonly only design and video processing is different in TV sets of different manufactures.

UPSample/Interpolation:

Upsampling is interpolation, applied in the context of digital signal processing and sample rate conversion. When upsampling is performed on a sequence of samples of a continuous function or signal, it produces an approximation of the sequence that would have been obtained by sampling the signal at a higher rate (or density, as in the case of a photograph). For example, if compact disc audio is upsampled by a factor of 5/4, the resulting sample-rate increases from 44,100 Hz to 55,125 Hz. Interpolation by an integer factor L can be explained as a 2-step process, with an equivalent implementation that is more efficient: When the interpolation filter is an FIR type, its efficiency can be improved, because the zeros contribute nothing to its dot product calculations. It is an easy matter to omit them from both the data stream and the calculations.

Edge preserving and smoothing:

Edge-preserving smoothing is an image processing technique that smooths away textures whilst retaining sharp edges. Examples are the Bilateral filter, the Guided filter and Anisotropic diffusion. n alternative to linear filtering, called anisotropic diffusion, was introduced by Perona and Malik. It is related to earlier work by Grossberg who used a similar nonlinear diffusion processes to model human vision. The motivation for anisotropic diffusion (also called nonuniform or variable conductance diffusion) is that a Gaussian smoothed image is a single time slice of the solution to the heat equation, that has the original image as its initial conditions. Anisotropic diffusion includes a variable conductance term that, in turn, depends on the differential structure of the image. Thus, the variable conductance can be formulated to limit the smoothing at “edges” in images, as measured by high gradient magnitude.

HMRF:

A hidden Markov random field is a generalization of a hidden Markov model. Instead of having an underlying Markov chain, hidden Markov random fields have an underlying Markov random field. The main difference with a hidden Markov model is that neighborhood is not defined in 1 dimension but within a network. the hidden Markov random field (HMRF) model and its expectation-maximization (EM) algorithm. We implement a MATLAB toolbox named HMRF-EM-image for 2D image segmentation using the HMRF-EM framework. This toolbox also implements edge-prior-preserving image segmentation, and can be easily reconfigured for other problems, such as 3D image segmentation.

Deconvolution:

In mathematics, deconvolution is an algorithm-based process used to reverse the effects of convolution on recorded data.^[1] The concept of deconvolution is widely used in the techniques of signal processing and image processing. Because these techniques are in turn widely used in many scientific and engineering disciplines, deconvolution finds many applications. finding the true PSF is impossible, and usually an approximation of it is used, theoretically calculated or based on some experimental estimation by using known probes. Real optics may also have different PSFs at different focal and spatial locations, and the PSF may be non-linear. The accuracy of the approximation of the PSF will dictate the final result. Different algorithms can be employed to give better results, at the price of being more computationally intensive. Since the original convolution discards data, some algorithms use additional data acquired at nearby focal points to make up some of the lost information. Regularization in iterative algorithms (as in expectation-maximization algorithms) can be applied to avoid unrealistic solutions. When the PSF is unknown, it may be possible to deduce it by systematically trying different possible PSFs and assessing whether the image has improved. This procedure is called blind deconvolution.

Superresolution:

Super-resolution imaging (SR) is a class of techniques that enhance the resolution of an imaging system. In some SR techniques—termed optical SR—the diffraction limit of systems is transcended, while in others—geometrical SR—the resolution of digital imaging sensors is enhanced. Super-resolution imaging techniques are used in general image processing and in super-resolution microscopy.Information When the term superresolution is used in techniques of inferring object details from statistical treatment of the image within standard resolution limits, for example, averaging multiple exposures, it involves an exchange of one kind of information (extracting signal from noise) for another (the assumption that the target has remained invariant). Resolution and localization True resolution involves the distinction of whether a target, e.g. a star or a spectral line, is single or double, ordinarily requiring separable peaks in the image. When a target is known to be single, its location can be determined with higher precision than the image width by finding the centroid (center of gravity) of its image light distribution. The word ultra-resolution had been proposed for this process[7] but it did not catch on, and the high-precision localization procedure is typically referred to as superresolution.

SYSTEM REQUIREMENTS:

HARDWARE REQUIREMENTS:

• System            : Pentium Dual Core.

• Hard Disk       : 120 GB.

• Monitor          : 15’’ LED

• Input Devices : Keyboard, Mouse

• Ram                : 1GB.

SOFTWARE REQUIREMENTS:

• Operating system : Windows 7.

• Coding Language : MATLAB

• Tool                       : MATLAB R2013A

REFERENCE:

Esmaeil Faramarzi, Member, IEEE, Dinesh Rajan, Senior Member, IEEE, Felix C. A. Fernandes, Member, IEEE, and Marc P. Christensen, Senior Member, IEEE, “Blind Super Resolution of Real-Life Video Sequences”, IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 25, NO. 4, APRIL 2016.