Blotches detection

51 HNUE JOURNAL OF SCIENCE Natural Sciences, 2020, Volume 65, Issue 4A, pp. 51-57 This paper is available online at BLOTCHES DETECTION Nguyen Thi Quynh Hoa Falcuty of Information Technology, Hanoi National University of Education Abstract. Blotch detection and removal is an important issue for archive film restoration and it can be extended in many other field of image processing. In this work, a new proposal including an automatic detection method and inpainting scheme is introduced. First, a technique for automatically detecting blotches based on local changes of pixels on consecutive frames is applied. Specifically, a two-stage Simplified Ranked Order Difference (SROD) detector is proposed to identify blotches on frames. Next, an improved inpainting was applied to restore the blotches so that it is undetectable by viewers. Our proposal is executed automatically without external parameters. The proposal has been tested on a serial of natural images with different sizes and resolutions. Experimental results show that the proposed solution has been successfully detected with fairly high accuracy and quite smooth restored blotches. Based on result analysis, the proposal has many potential and applications in the future. Index Terms-blotch detection, blotch removal, restoration, inpainting. Keywords: Terms-blotch detection, blotch removal, restoration, inpainting. 1. Introduction Storage and processing of visual aids that do not affect their overall quality is still a challenging problem. Storage and handling of old movies is even more difficult due to the physicochemical properties of the film. Indeed, many physical factors can affect the overall quality of the old-time movie such as moisture and heat, one of the factors responsible for the fading of old movies. One of the common issues of interest and research is the restoration of old films damaged by various problems like grainy, fuzzy, faded and artifacts other structures. Until now, a considerable effort has been devoted to developing methods and systems of digital precision to restore old film. The stretch marks and scratches are some artifacts film is the most popular study [1-2]. During long time ago, the old film was restored by one of the two most common techniques that technical manual or semi-automatic. However, the quality of the recovery depends on the skill of the user and the limitations of the tools and methods used. For all these reasons, the film industry is always searching and researching the advantages of digital solutions. Some techniques are proposed based on the model of film degradation phenomena [3]. However, these models often ignore the cognitive aspect and therefore excludes the human observer, the final judge. Various methods for removing spots have been proposed in the literature. Some methods based on image printing techniques [4-5]. In addition, a simple way Received March 20, 2020. Revised May 7, 2020. Accepted May 14, 2020 Contact Nguyen Thi Quynh Hoa, e-mail address: hoantq@hnue.edu.vn Nguyen Thi Quynh Hoa 52 to expand the method to blur images into videos however it can lead to incoherent results [6] and especially around moving objects. The purpose of this work we propose a complete solution for detecting and removing spots movies based on some perceived feature of the human visual system. The performance of each component is assessed through objective research extensively tested on real video. The perceived quality of the frames has been restored and are evaluated by fuzzy picture quality printing for professional use. 2. Content 2.1. Monoresolution detection of suspicious regions An input image I(k) could be easily represented at different resolution levels by a Gaussian pyramid. The spots are treated like a change of lighting locally between the degraded image I(k) and every reference I(k-1) and I(k+1). However, earlier methods are not handling the different size of the spot as well as the shift of different intensity, because it works directly on the resolution of the input image. This leads to reduce the incidence of false alarms and improve detection accuracy rate [7]. The spot detection includes the two main steps. First identify likely areas placed first broken. Then, a heuristic detection for SROD only machine is applied to those areas. More precisely, the suspicious region is positioned using a powerful movement patterns to lighting variations. In this respect, we retain the affine motion model proposed in [8] for efficiency and simplicity of it. Taking into consideration the displacement dj(k-1,k) (rj) of the pixel rj between Aj(k-1) and Aj(k), and the illumination coefficient hj(k-1,k) that reflects its intensity change between both sub-images, the observation model is defined by ܣ௝ ሺ௞ሻ(ݎ௝) ൌ ℎ௝ ሺ௞−ଵ,௞ሻ(ݎ௝)ܣ௝ ሺ௞−ଵሻ ൬ݎ௝ ൅ ௝݀ ሺ௞−ଵ,௞ሻ(ݎ௝)൰ ൅ ௝ܾ ሺ௞−ଵ,௞ሻ(ݎ௝) (1) where ௝ܾ ሺ௞−ଵ,௞ሻ is the estimation error. As these parameters are locally valid, they are estimated in blockwise way by ther Block-Matching algorithm (BMA). Thus, the sub-image Aj(k) is firstly partitioned into blocks ܤሺ௨ೕ,௩ೕሻ ሺ௞ሻ of size ℓ௝ × ℓ௝ ∈  [1, , ௅భ 2ೕ] × [1, , ௅మ 2ೕ]. Then, the optimal block ܤሺ௨ೕ∗,௩ೕ∗ሻ ሺ௞−ଵሻ in Ǎ௝ ሺ௞−ଵሻ is the one that minimizes the mean squared error relatively to the considered blockܤሺ௨ೕ,௩ೕሻ ሺ௞ሻ in a searche area ܵሺ௨ೕ,௩ೕሻ ሺ௞−ଵሻ : The illumination parameters are given by ∀ݎ௝ ∈ ܤ(௨ೕ,௩ೕ) ሺ௞ሻ , ĥ(௨ೕ,௩ೕ) ሺ௞−ଵ,௞ሻ(ݎ௝) ൌ ஼ೀೇሺ஻ቀೠೕ,ೡೕቁ ሺೖሻ ,஻̌ ቀೠೕ , ,ೡೕ , ቁ ሺೖషభሻ ఙ ಳෙ ൬ೠೕ , ,ೡೕ , ൰ ሺೖషభሻ మ  (2) and  ෠ܾ(௨ೕ,௩ೕ) ሺ௞−ଵሻ (ݎ௝) ൌ ݉஻ ቀೠೕ,ೡೕቁ ሺೖሻ െ ĥ(௨ೕ,௩ೕ) ሺ௞−ଵ,௞ሻ(ݎ௝)݉஻̌ ቀೠೕ , ,ೡೕ , ቁ ሺೖషభሻ . (3) Since most of the reported statistical tests require that analyzed data be normally distributed, the set ܪሺ௝ሻ ሺ௞−ଵ,௞ሻ ൌ ቄℎ(௨ೕ,௩ೕ) ሺ௞−ଵ,௞ሻቅ (௨ೕ,௩ೕ) is firstly transformed by using the Box-Cox transformation [9]. This transformation give rise to the set ܪ෩ሺ௝ሻ ሺ௞−ଵ,௞ሻ ൌ ቄℎ̃(௨ೕ,௩ೕ) ሺ௞−ଵ,௞ሻቅ (௨ೕ,௩ೕ) normally distributed. Then, in order to locate the outliers within ܪ෩ሺ௝ሻ ሺ௞−ଵ,௞ሻ, it is possible to apply one statistical test among the ones reported [10-11]. In this work, we retain the Minimum Blotches detection 53 Covariance Determinant (MCD) test for its efficiency and relatively low computational complexity [10]. The MCD test provides a set  ሺܱ௝ሻ ሺ௞−ଵ,௞ሻ of a typical values of the illumination coefficient that are asssociated to candidate blocks in Aj(k) and Aj(k-1) and are more susceptible to be blotched. The same procedure is carried out to detect suspicious blocks between Aj(k) and Aj(k+1). Only the blocks that are judged as suspicious in both the backward and the forward directions (by considering respectively the pairs (Aj(k-1), Aj(k)) and (Aj(k+1), Aj(k)) are retained for the final blotch detection step since blotches are considered as illumination variations occuring in both directions. 2.2. Blotch detection Given the positions of the suspicious regions at different resolution levels, it is necessary to deduce the positions of the retained candidate regions at the initial resolution level. Since the goal behind resorting to a multiscale analysis is to handle the different sizes of the blotches, we consider a block at the initial resolution level as suspicious if it has been judged as such at, at least one resolution level initial resolution level. Since the goal behind resorting to a multiscale analysis is to handle the different sizes of the blotches, we consider a block at the initial resolution level as suspicious if it has been judged as such at, at least one resolution level ݆, ∀1 ≤ ݆ ≤ ܬ. The final step consists of detecting the corrupted pixels in each candidate region at the initial resolution of ܫሺ௞ሻ. For this purpose, one of the reported heuristic detectors as the SDIa, the ROD, the SROD, or the AR based detectors is used between suspicious blocks in ܫሺ௞ሻand their homologous in the motion compensated refrence frames. 2.3. Blotch removal We assume that the spots are identified and detected correctly in the previous step at this stage is a binary table, where the mottled pixel is color coded white and the rest black. This binary image is used to guide the recovery process. To remove blotches are detected, an approach based on the proposed inpainting. The performance of the solution is assessed subjectively or using some full reference image quality classical metrics like PSNR. In this work, we use hierarchical diagram similar to [12] and strategic global optimization. This approach introduces an effective performance and high quality output. (a) A blotched image (b) bloctch detection map (c) The initial priority map Figure 1. An example of the priority map Nguyen Thi Quynh Hoa 54 Firstly, a Gaussian pyramid was built from the original image to create hierarchical diagrams of input images. A set of images with different levels of detail can be created with ܩ଴ ൌ ܫ as the input or original image. Number of pyramid levels depending on the original size of the image and the minimum resolution allowed. Then, a strategy used to fill in missing areas with the lowest resolution, ܩே. Athigher resolutions, inpainting problem are modeled as a graph optimization labels which help to show the selected label for each pixel unknown. It can be determined by optimizing the energy function optimization algorithm by global, multi- labeled graph cut [13]. A description of the algorithm removes spots is given in Figure 1. 2.3.1. Correction of lowest resolution frame At lowest resolution, a template-based approach is used to remove the blob by using priority based on the window and choose the patch. Recommended removal methods work repeated as follows: - Detect spots: Identify spots and their boundaries based on the binary image. If no pixels in the spot, the algorithm is terminated due spots completely erased. - Define priorities edit: Calculator and randomly select a pixel p with the highest priority and determines a patch, P, gathered at p. In this work, we used the model of the most common priorities proposed in [14, 15] Combine the patch: Find a patch is not blurred, ߰௤, similar to ߰௣ with mean squared error of pixel squares is not blurred. - Blotch removal: Fill in missing information in patch ߰௤. - Information update: Update the binary mask image and return the step 1. Correction priority: So good priorities is essential because it directly affects the quality of the output. In this work, we used the model of the most common priorities proposed in [15] and it is defined as in Equation 4 ሺܲ௣ሻ ൌ ܥሺ௣ሻܦሺ௣ሻ (4) where C(p) and D(p) denote confidence and data terms, respectively and they are defined as follows: ܥሺ௣ሻ ൌ ∑ ஼ሺ೜ሻ೜ചಇ೛∩ಈ |ஏ೛| (5) ܦሺ݌ሻ ൌ ఒభఒమ+ఢ (6) During the initialization process, the values of reliability, C(p) is set to 0 for each pixel in blotched and 1 for others. ∈ is very small positive value, which ensures that terminology is always dominate the other. ߣଵ and ߣ2 are two positive eigenvalues (ߣଵ≥ ߣ2 ) determine the local changes of pixel intensities in each window Wp, gathered at p and is characterized by the following matrix: ܯሺ௣ሻ ൌ ∑ܩௐ೛ሺݔ, ݕሻ൮ ቀడூడ௫ቁ 2 డூ డ௫ డூ డ௬ డூ డ௫ డூ డ௬ ቀ డூ డ௬ቁ 2൲ (7) where ܩௐ೛ is a Gaussian window function calculates a total weight. The term data including structural features depends on the variation of two separate values. Figure 1 illustrates a preferred map said pixels will be restored first. Patch selection: The next step in the optimization algorithm is searching for patches matching the blurred area. In our work, a suitable patch is determined using similar Blotches detection 55 measurements on all pixels not faded in patches. Therefore, it is determined based on the difference of color and gradient as below: ݀(Ψ௣,Ψ௤) ൌ ∑ ቀ(ܫ௣௜ െ ܫ௤௜ ) 2 ൅ ߠ(∇ܫ௣௜ െ ∇ܫ௤௜ ) 2ቁ (8) where Ip, Iq are the corresponding CIELab vectors; ∇ܫ௣, ∇ܫ௤ represent the image gradient vectors. ߠ is a user defined weight balancing the two terms. In our experiments, we used ߠ ൌ 0.5. The patch with minimal distance to the source patch, ߰௣, is the chosen one and given below: ߰௣̂ ൌ ܽݎ݃݉݅݊ట∈Φ{݀(Ψ௣,Ψ௤)} (9) 2.3.2. Correction for higher resolution frame When finished creating images lowest resolution, compensation map is generated and used to reconstruct a higher resolution. Map offset determine the relationship between the pixel needs to be removed and the pixels in the region are not blurred are given below ܵܯሺ௣ሻ ൌ ቄሺ଴,଴ሻ௢௧௛௘௥௪௜௦௘ ∆௫,∆௬௣ሺ௫,௬ሻ∈Ω (10) Compensation map obtained from the lower resolution is interpolated to higher resolution. However, the output image is derived directly from this map which contains annoying artifacts affect the nature of the images obtained. The authors of [16] the data and smoothness to refine compensation map. Energy function is defined as follows: Ḃ(௨ೕ∗,௩ೕ∗) ሺ௞−ଵሻ ൌ ܽݎ Ḃ ቀೠೕ ′,ೡೕ ′ቁ ሺೖషభሻ ∈ௌ ቀೠೕ,ೡೕቁ ሺೖషభሻ ݉݅݊∑ ሺܣ௝ ሺ௞ሻ(ݎ௝) െ ℎ(௨ೕ,௩ೕ) ሺ௞−ଵ,௞ሻ ௥ೕ∈஻ቀೠೕ,ೡೕቁ ሺೖሻ Ǎ௝ ሺ௞−ଵሻ ൬ݎ௝ ൅ ௝݀ ሺ௞−ଵ,௞ሻ(ݎ௝)൰ െ ܾ(௨ೕ,௩ೕ) ሺ௞−ଵሻ ሻ2 (11) ܧܯ ൌ ߙ∑ܧௗ(ܵܯሺ௣ሻ) ൅ ሺ1 െ ߙሻ∑ ܧ௦ሺܵܯሺ௣ሻ, ܵܯሺ௤ሻሻሺ௣,௤ሻ∈ே஻ (12) where ܧௗ is a data term related to external requirements and ܧ௦ is a smoothness term defined over a set of neighbouring pixels, N B. The parameter a is a user defined weight balancing the twotermssettoα=0.5inourexperiment.The detail of the data term and smoothness term are given by equation (12), (13): ܧௗሺܵܯሺ௣ሻሻ ൌ ቄ ௢௧௛௘௥௪௜௦௘ ሺ௫+∆௫,௬+∆௬ሻ∈Ω ଴ ∞ (13) whereβandγareweightsbalancingthesetwoterms,settoβ=l,γ=2inourexperiment.There are many approaches for minimizing this function. In the proposed method, a global optimization based on graph-cuts is developed because of the efficient implementation and the available source code. The alpha parameter is to determine the importance, or the balance between two operands. In this case, Ed and Es are chosen as 0.5 because these two operands are considered to have the same role, so they are equal. 2.4. Experiment results This section is dedicated to the performance evaluation of the proposed framework. The performance of blotch detector scheme is evaluated in three round of simulation. The first round shows the benefits of versatile analysis versus monochrome case as in [7]. The second round presents a visual assessment using SROD detection in locating the damaged area candidates. And the last round is to evaluate the performance in comparison with the proposal in [17]. ܧ௦ሺܵܯሺ௣ሻ, ܵܯሺ௤ሻሻ ൌ ൜ ௢௧௛௘௥௪௜௦௘ ௌ ሺ೛ሻ=ௌெሺ೜ሻ ఉఋெ(ௌெሺ೛ሻ)+ఊఋீሺௌெሺ೛ሻሻ ଴ (14) A sequence of thirteen frames has been used for testing. Frames are 720 × 576 pixels and encoded, which shows some typical frames to provide a visual demonstration of our proposal. Nguyen Thi Quynh Hoa 56 The first line contains some corrupted frames and the blotch detection is detected and represented by binary mask frames of the second line. The blotch removal frames are shown in the last line. In order to evaluate of performance the blotch removal algorithm, we compared with some state-of-the-art inpainting methods such as A. Criminisi et al. [14], Wu et al. [18] and Dang et al. [15]. The image quality of the proposed method has been objective evaluation with unclear quality indicators [19] and shown in Table 1. The higher the result are, the better the quality of the propose approach is. Table 1. The blotch removal quality metric Method Our proposal [15] [14] [18] Frame ID 18 0.2159 0.199 0.2118 0.2182 20 0.1987 0.2 0.2022 0.2032 32 0.1178 0.1105 0.1234 0.1095 37 0.1391 0.1248 0.1286 0.1321 46 0.1078 0.1075 0.1102 0.1135 50 0.1735 0.1724 0.1705 0.1773 68 0.1711 0.1591 0.1577 0.1727 85 0.1928 0.182 0.1825 0.1844 94 0.1934 0.1934 0.1946 0.1939 102 0.2382 0.2436 0.242 0.2367 103 0.3363 0.3456 0.3422 0.3227 111 0.2382 0.2436 0.242 0.2367 118 0.1429 0.1399 0.1395 0.1418 3. Conclusions The article has proposed a framework to detect and restore spots. The framework consists of two main stages: detection and recovery. For spot detection, a simple rank order difference (SROD) detector is proposed. Next, spots will be restored based on improved dimming printing techniques. The test results show an outstanding performance and expected results. The overall execution time of the schema is completely acceptable. In the future, we will improve and upgrade the framework to make it better with larger resolution videos. REFERENCES [1] T. Hoshi, T. Komatsu, and T. Saito, 1988. Film blotch removal with a spatiotemporal fuzzy filter based on local image analysis of anisotropic continuity . Int. Conf. Image Process, pp. 478-482. [2] P. M. B. V. Roosmalen, 1999. Restoration of archived film and video. Ph.D. dissertation, Delft University of Technology, The Netherlands. Blotches detection 57 [3] Q. Do, A. Beghdadi, and M. Luong, 2013. Color mismatch compensation method based on a physical model. IEEE Transactions on Circuits and Systems for Video Technology, Vol. 3, pp. 244-257. [4] A. Gangal and B. Dizdaroglu, 2006. Automatic restoration of old motion picture films using spatiotemporal exemplar-based inpainting. Advanced Concepts for Intelligent Vision Systems, Vol. 4179, pp. 55-66. [5] W. Yan and M. Kankanhalli, 2002. Erasing video logos based on image inpainting. IEEE International Conference on Multimedia and Expo (ICME 2002), Vol. 2, pp. 521-524, [6] Y. Wexler, E. Shechtman, and M. Irani, 2007. Space-time completion of video. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, Vol. 29, pp. 463-476. [7] H. Ammar-Badri and A. Benazza-Benyahia, 2011. Improving blotch detection in old films by a preprocessing step based on outlier statistical test. IEEE EUropean SIgnal Processing COnference, Barcelona, Spain, [8] F. J. Hampson and J. C. Pesquet, 2000. Motion estimation in the presence of illumination variations. Signal processing: Image communication, Vol. 16, No. 4, pp. 373-381. [9] E. P. G. Box and D. R. Cox, 1964. An analysis of transformations. Journal of Royal Statistical Society, Series B (Methodological), Vol. 26, pp. 211-252. [10] P. J. Rousseeuw and K. V. Driessen, 1999. A fast algorithm for the minimum covariance determinant estimator. American Statistical Association and the American Society for Quality, Vol. 41, No. 3, pp. 212-223. [11] M. Riani, A. C. Atkinson, and A. Cerioli, 2009. Finding an unknown number of multivariate outliers. Journal of the Royal Statistical Society: series B (statistical methodology), Vol. 71, No. 2, pp. 447-466. [12] D. T. Trung, A. Beghdadi, and M.-C. Larabi, 2014. A perceptual image completion approach based on a hierarchical optimization scheme. Signal Processing, Vol. 103, pp. 127-141. [13] Y. Boykov, O. Veksler, and R. Zabih, 2001. Fast approximate energy min-imization via graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 23, pp. 1222-1239. [14] A. Criminisi, P. Perez, and K. Toyama, 2004. Region filling and object removal by exemplar-based image inpainting. IEEE Transaction of Image Process, Vol. 13(9), pp. 1200-1212. [15] T. T. Dang, M. C. Larabi, and A. Beghdadi, 2012 . Multi-resolution patch and window- based priority for digital image inpainting problem. 3rd International Conference on Image Processing Theory, Tools and Ap- plications, pp. 280-284. [16] A. Agarwala, M. Dontcheva, M. Agrawala, S. Drucker, A. Colburn, B. Curless, D. Salesin, and M. Cohen, 2004. Interactive digital photomon-tage. Proceedings of SIGGRAPH, pp. 294-302. [17] M. K. Gullu, O. Urhan, and S. Erturk, 2008. Blotch detection and removal for archive film restorati