Color in Image and Video Processing-Most Recent Trends and Future Research Directions

The motivation of this paper is to provide an overview of the most recent trends and of the future research directions in color image and video processing. Rather than covering all aspects of the domain this survey covers issues related to the most active research areas in the last two years. It presents the most recent trends as well as the state-of-the-art, with a broad survey of the relevant literature, in the main active research areas in color imaging. It also focuses on the most promising research areas in color imaging science. This survey gives an overview about the issues, controversies, and problems of color image science. It focuses on human color vision, perception, and interpretation. It focuses also on acquisition systems, consumer imaging applications, and medical imaging applications. Next it gives a brief overview about the solutions, recommendations, most recent trends, and future trends of color image science. It focuses on color space, appearance models, color difference metrics, and color saliency. It focuses also on color features, color-based object tracking, scene illuminant estimation and color constancy, quality assessment and fidelity assessment, color characterization and calibration of a display device. It focuses on quantization, filtering and enhancement, segmentation, coding and compression, watermarking, and lastly on multispectral color image processing. Lastly, it addresses the research areas which still need addressing and which are the next and future perspectives of color in image and video processing.

1. Background and Motivation

The perception of color is of paramount importance in many applications, such as digital imaging, multimedia systems, visual communications, computer vision, entertainment, and consumer electronics. In the last fifteen years, color has been becoming a key element for many, if not all, modern image and video processing systems. It is well known that color plays a central role in digital cinematography, modern consumer electronics solutions, digital photography system such as digital cameras, video displays, video enabled cellular phones, and printing solutions. In these applications, compression- and transmission-based algorithms as well as color management algorithms provide the foundation for cost effective, seamless processing of visual information through the processing pipeline. Moreover, color also is crucial to many pattern recognition and multimedia systems, where color-based feature extraction and color segmentation have proven pertinent in detecting and classifying objects in various areas ranging from industrial inspection to geomatics and to biomedical applications.

Over the years, several important contributions were made in the field of color image processing. It is only since the last decades that a better understanding of color vision, colorimetry, and color appearance has been utilized in the design of image processing methodologies [1]. The first special issue on this aspect was written by McCann in 1998 [2]. According to McCann, the problem with display devices and printing devices is that they work one pixel at a time, while the human visual system (HSV) analyzes the whole image from spatial information. The color we see at a pixel is controlled by that pixel and all the other pixels in the field of view [2]. In our point of view, the future of color image processing will pass by the use of human vision models that compute the color appearance of spatial information rather than low level signal processing models based on pixels, but also frequential, temporal information, and the use of semantic models. Human color vision is an essential tool for those who wish to contribute to the development of color image processing solutions and also for those who wish to develop a new generation of color image processing algorithms based on high-level concepts.

A number of special issues, including survey papers that review the state-of-the-art in the area of color image processing, have been published in the past decades. More recently, in 2005 a special issue on color image processing was written for the signal processing community to understand the fundamental differences between color and grayscale imaging [1]. In the same year, a special issue on multidimensional image processing was edited by Lukac et al. [3]. This issue overviewed recent trends in multidimensional image processing, ranging from image acquisition to image and video coding, to color image processing and analysis, and to color image encryption. In 2007, a special issue on color image processing was edited by Lukac et al. [4] to fill the existing gap between researchers and practitioners that work in this area. In 2007, a book on color image processing was published to cover processing and application aspects of digital color imaging [5].

Several books have also been published on the topic. For example, Lukac and Plataniotis edited a book [6] which examines the techniques, algorithms, and solutions for digital color imaging, emphasizing emerging topics such as secure imaging, semantic processing, and digital camera image processing.

Since 2006, we have observed a significant increase in the number of papers devoted to color image processing in the image processing community. We will discuss in this survey which are the main problems examined by these papers and the principal solutions proposed to face these problems. The motivation of this paper is to provide a comprehensive overview of the most recent trends and of the future research directions in color image and video processing. Rather than covering all aspects of the domain, this survey covers issues related to the most active research areas in the last two years. It presents the most recent trends as well as the state-of-the-art, with a broad survey of the relevant literature, in the main active research areas in color imaging. It also focuses on the most promising research areas in color imaging science. Lastly, it addresses the research areas which still need addressing and which are the next and future perspectives of color in image and video processing.

This survey is intended for graduate students, researchers and practitioners who have a good knowledge in color science and digital imaging and who want to know and understand the most recent advances and research in digital color imaging. This survey is organized as follows: after an introduction about the background and the motivation of this work, Section 2 gives an overview about the issues, controversies, and problems of color image science. This section focuses on human color vision, perception, and interpretation. Section 3 presents the issues, controversies, and problems of color image applications. This section focuses on acquisition systems, consumer imaging applications, and medical imaging applications. Section 4 gives a brief overview about the solutions, recommendations, most recent trends and future trends of color image science. This section focuses on color space, appearance models, color difference metrics, and color saliency. Section 5 presents the most recent advances and researches in color image analysis. Section 5 focuses on color features, color-based object tracking, scene illuminant estimation and color constancy, quality assessment and fidelity assessment, color characterization and calibration of a display device. Next, Section 6 presents the most recent advances and researches in color image processing. Section 6 focuses on quantization, filtering and enhancement, segmentation, coding and compression, watermarking, and lastly on multispectral color image processing. Finally, conclusions and suggestions for future work are drawn in Section 7.

2. Color Image Science at Present: Issues, Controversies, Problems

2.1. Background

The science of color imaging may be defined as the study of color images and the application of scientific methods to their measurement, generation, analysis, and representation. It includes all types of image processing, including optical image production, sensing, digitalization, electronic protection, encoding, processing, and transmission over communications channels. It draws on diverse disciplines from applied mathematics, computing, physics, engineering, and social as well as behavioural sciences, including human-computer interface design, artistic design, photography, media communications, biology, physiology, and cognition.

Although digital image processing has been studied for some 30 years as an academic discipline, its focus in the past has largely been in the specific fields of photographic science, medicine, remote sensing, nondestructive testing, and machine vision. Previous image processing and computer vision research programs have primarily focused on intensity (grayscale) images. Color was just considered as a dimensional extension of intensity dimension, that is, color images were treated just as three gray-value images, not taking into consideration the multidimensional nature of human color perception or color sensory system in general. The importance of color image science has been driven in recent years by the accelerating proliferation of inexpensive color technology in desktop computers and consumer imaging devices, ranging from monitors and printers to scanners and digital color cameras. What now endows the field with critical importance in mainstream information technology is the very wide availability of the Internet and World Wide Web, augmented by CD-ROM and DVD storage, as a means of quickly and cheaply transferring color image data. The introduction of digital entertainment systems such as digital television and digital cinema required the replacement of the analog processing stages in the imaging chain by digital processing modules, opening the way for the introduction to the imaging pipeline of the speed and flexibility afforded by digital technology. The convergence of digital media, moreover, makes it possible for the application of techniques from one field to another, and for public access to heterogeneous multimedia systems.

For several years we have been facing the development of worldwide image communication using a large variety of color display and printing technologies. As a result, “cross media” image transfer has become a challenge [7]. Likewise, the requirement of accuracy on color reproduction has pushed the development of new multispectral imaging systems. The effective design of color imaging products relies on a range of disciplines, for it operates at the very heart of the human-computer interface, matching human perception with computer-based image generation.

Until recently, the design of efficient color imaging systems was guided by the criterion that “what the user cannot see does not matter.” This is no longer true. This has been, so far, the only guiding principle for image filtering and coding. In modern applications, this is not sufficient enough. For example, it should be possible to reconstruct on display the image of a painting from a digital archive under different illuminations. From the human vision point, the problem is that visual perception is one of the most elusive and changeable of all aspects of human cognition, and depends on a multitude of factors. Successful research and development of color imaging products must therefore combine a broad understanding of psychophysical methods with a significant technical ability in engineering, Modern heuristic techniques for combinatorial problems. Advanced topics in computer science
free download

Decision Tree and Ensemble Learning Based on Ant Colony Optimization Google Books Result Modern heuristic techniques for combinatorial problems . Faculty library sciences Series: Advanced topics in computer science series; Alternative call (PDF) Modern Heuristic Search Methods

Software Agents Michael R. Genesereth Logic Group Computer Science Department Stanford University
free download

The software world is one of great richness and diversity. Many thousands of software products are available to users today, providing a wide variety of information and services in a wide variety of domains. While most of these programs provide their users with significant Classical complexity theory analyzes and classifies problems by the amount of a resource, usually time or space, that is required by algorithms solving them. It was a fundamental idea, going back to the work of Hartmanis and Stearns in the early 1960s, to measure the required

The combinatorics of network reliability, International Series of Monographs on Computer Science
free download

This book develops combinatorial tools which are useful for reliability analysis, as demonstrated with a probabilistic network model. Basic results in combinatorial enumeration are reviewed, along with classical theorems on connectivity and cutsets. More developed

Languages and machines: an introduction to the theory of computer science
free download

Languages and machines : an introduction to the theory of computer . Theory of Computing 2015/2016 (FUB MSc in Computer Science ) Languages and Machines, which is intended for computer scientists in the theoretical foundations of their subject, gives a mathematically sound In the late 1960s and early 1970s, the concepts of graph transformation and graph grammars started to become of interest in picture processing and computer science . The main idea was to generalize well-known rewriting techniques from strings and trees to Parsing is the process of structuring a linear representation in accordance with a given grammar. This definition has been kept abstract on purpose to allow as wide an interpretation as possible. The linear representation may be a sentence, a computer

The Cambridge distributed computing system. International computer science series
free download

The Development of Component-based Information Systems Google Books Result The Cambridge Distributed Computing System is an early discontinued distributed operating system, . (International computer science series) Bibliography: p. The 100 Best Computer Science Programs in the

Logic and the challenge of computer science
free download

Nowadays computer science is surpassing mathematics as the primary field of logic applications, but logic is not tuned properly to the new role. In particular, classical logic is preoccupied mostly with infinite static structures whereas many objects of interest in

Computer science unplugged
free download

For kids ages 7 to 14: This educational 50 minute show is an action-packed, zany time exploring neat ideas in computer science . It has been performed for over 20 years in classrooms, science museums, science festivals, and at educational events. Kids, and the

The Profession of IT, Is Computer Science Science
free download

COMMUNICATIONS OF THE ACM draw on the same fundamental principles. In 1989, we used the term computing instead of computer science , mathematics, and engineering. Today, computing science, engineering, mathematics, art

Discrete mathematical structures with applications to computer science
free download

The objectives of the course are: To develop Professional Skills through effective communication To introduce a number of Mathematical Foundation to be serving as tools even today in the development of theoretical computer science To gain some confidence on

Bringing computational thinking to K-12: what is Involved and what is the role of the computer science education community
free download

When Jeanette Wing [13] launched a discussion regarding the role of computational thinking across all disciplines, she ignited a profound engagement with the core questions of what computer science is and what it might contribute to solving problems across the

Modern DC-to-DC Switchmode Power Converter Circuits (Van Nostrand Reinhold Electrical/ Computer Science and Engineering Series)
free download

As each area of technology with a potential for significantly impacting any major segment of the electronics industry evolves, it often is accompanied by the development of a succession of new circuits. Each new circuit indeed appears different, employing different components

Information technology research: A practical guide for computer science and informatics
free download

Information Technology Research: A Practical Guide for Computer

Scientific methods in computer science
free download

ABSTRACT This paper analyzes scientific aspects of Computer Science . First it defines science and scientific method in

Active learning and its use in computer science
free download

Student learning and the depth of the students knowledge increase when active learning methods are employed in the classroom. Active learning strategies are discussed in general computer science course work and as used in a theory of computation course. Difficulties

Why the high attrition rate for computer science students: some thoughts and observations
free download

1. Introduction At our university, there are over four hundred declared majors in Computer Science . Each semester, however, only about fifteen to twenty students graduate in this field. The freshman courses comprise overflowing multiple sections, but the upper level courses make

Form and content in computer science
free download

The trouble with computer science today is an obsessive concern with form instead of content. No, that is the wrong way to begin. By any previous standard the vitality of computer science is enormous; what other intellectual area ever advanced so far in twenty years

Why women avoid computer science
free download

COMMUNICATIONS OF THE ACM did these numbers drop, and why more sharply for women than for men For men, the explanation is obvious. Traditional paths to wealth like law, medicine, and business are more certain, and over the
, applied mathematics, and behavioral science.

2.2. Human Color Vision

The human color vision system is immensely complicated. For a better understanding of its complexity, a short introduction is given here. The reflected light from an object enters the eye, first passes through the cornea and lens, and creates an inverted image on the retina at the back of the eyeball. The retinal surface contains millions of two types of photoreceptors: rods and cones. The former are sensitive to very low levels of light but cannot see color. Color information is detected at normal (daylight) levels of illumination by the three types of cones, named L, M, S, corresponding to light sensitive pigments at long, medium, and short wavelengths, respectively. The visible spectrum ranges between about 380 to 780 nanometers (nm). The situation is complicated by the retinal distribution of the photoreceptors: the cone density is the highest in the foveal region in a central visual field of approximately 2° diameter, whereas the rods are absent from the fovea but attain maximum density in an annulus of 18° eccentricity, that is, in the peripheral visual field. The information acquired by rods and cones is encoded and transmitted via the optic nerve to the brain as one luminance channel (black-white) and two opponent chrominance channels (red-green and yellow-blue), as proposed by the opponent-process theory of color vision of Hering. These visual signals are successively processed in the lateral geniculate nucleus (LGN) and visual cortex (V1), and then propagated to several nearby visual areas in the brain for further extraction of features. Finally, the higher cognitive functions of object recognition and color perception are attained.

At very low illumination levels, when the stimulus has a luminance lesser than approximately 0.01 cd/m2, only the rods are active and give monochromatic vision, known as scotopic vision. When the luminance of the stimulus is greater than approximately 10 cd/m2, at normal indoor and daylight level of illumination in a moderate surround, the cones alone mediate color vision, known as photopic vision. In between 0.01 and 10 cd/m2 there is a gradual changeover from scotopic to photopic vision as the retinal illuminance increases, and in this domain of mesopic vision both cones and rods make significant contributions to the visual response.

Yet the mesopic condition is commonly encountered in dark-surround or dim-surround conditions for viewing of television, cinema, and conference projection displays, so it is important to have an appropriate model of color appearance. The cinema viewing condition is particularly interesting, because although the screen luminance is definitely photopic, with a standard white luminance of 40–50 cd/m2, the observers in the audience are adapted to a dark surround in the peripheral field which is definitely in the mesopic region. Also, the screen fills a larger field of view than is normal for television, so the retinal stimulus extends further into the peripheral field where rods may make a contribution. Additionally, the image on the screen changes continuously and the average luminance level of dark scenes may be well down into the mesopic region. Under such conditions, the rod contribution cannot be ignored. There is no official CIE standard yet available for mesopic photometry, although in Division 1 of the CIE there is a technical committee dedicated to this aspect of human vision: TC1-58 “Visual Performance in the Mesopic Range.”

When dealing with the perception of static and moving images, visual contrast sensitivity plays an important role in the filtering of visual information processed simultaneously in the various visual “channels.” The high frequency active channels (also known as parvocellular or P channels) enable detail perception; the medium frequency active channels allow shape recognition, whereas the low-frequency active channels (also known as magnocellular or M channels) are more sensitive to motion. Spatial contrast sensitivity functions (CSFs) are generally used to quantify these responses and are divided into two types: achromatic and chromatic. Achromatic contrast sensitivity is generally higher than chromatic. For achromatic sensitivity, the maximum sensitivity to luminance for spatial frequencies is approximately 5 cycles/degree. The maximum chrominance sensitivity is only about one tenth of the maximum luminance sensitivity. The chrominance sensitivities fall off above 1 cycle/degree, particularly for the blue-yellow opponent channel, thus requiring a much lower spatial bandwidth than luminance. For a nonstatic stimulus, as in all refreshed display devices, the temporal contrast sensitivity function must also be considered. To further complicate matters, the spatial and temporal CSFs are not separable and so must be investigated and reported as a function on the time-space frequency plane.

Few research groups have been working on the mesopic domain; however there is a need for investigation. For example, there is a need to develop metrics for perceived contrasts in the mesopic domain [8]. In 2005, Walkey et al. proposed a model which provided insight into the activity and interactions of the achromatic and chromatic mechanisms involved in the perception of contrasts [9]. However, the proposed model does not offer significant improvement over other models in high mesopic range or in mid-to-low mesopic range because the mathematical model used is not relevant to adjust correctly these extreme values.

Likewise, there is a need to determine the limits of visibility, for example, the minimum of brightness contrast between foreground and background, in different viewing conditions. For example, Ojanpaa et al. investigated the effect of luminance and color contrasts on the speed of reading and visual search in function of character sizes. It would be interesting to extend this study to small displays such as mobile devices and to various viewing conditions such as under strong ambient light. According to Kuang et al., contrast judgement as well as colorfulness has to be analysed in function of highlight contrasts and shadow contrasts [10].

2.3. Low-Level Description and High-Level Interpretation

In recent years, research efforts have also focused on semantically meaningful automatic image extraction [11]. According to Dasiapoulou et al. [11], these efforts have not bridged the gap between low-level visual features that can be automatically extracted from visual content (e.g., with saliency descriptors), and the high-level concepts capturing the conveyed meaning. Even if conceptual models such as MPEG7 have been introduced to model high-level concepts, we are always confronted to the problem of extracting the objects of a scene (i.e., the regions of an image) at intermediate level between the low level and the high level. Perhaps the most promising way to bridge the former gap is to focus the research activity on new and improved human visual models. Traditional models are based either on a data-driven description or on a knowledge-based description. Likewise, there is in a general way a gap between traditional computer vision science and human vision science, the former considering that there is a hierarchy of intermediate levels between signal-domain information and semantic understanding meanwhile the latter consider that the relationships between visual features in the human visual system are too complex to be modeled by a hierarchical model. Alternative models attempted to bridge the gap between low-level descriptions and high-level interpretations by encompassing a structured representation of objects, events, relations that are directly related to semantic entities. However, there is still plenty of space for new alternative models, additional descriptors and methodologies for an efficient fusion of descriptors [11].

Image-based models as well as learning-based approaches are techniques that have been widely used in the area of object recognition and scene classification. They consider that humans can recognize objects either from their shapes or from their color and their texture. This information is considered as low-level data because it is extracted by the human vision system during the preattentive stage. Inversely, high-level data (i.e., semantic data) is extracted during the interpretation stage. There is no consensus in human vision science to model intermediate stages between preattentive and interpretation stages because we do not have a complete knowledge of visual areas and of neural mechanisms. Moreover, the neural pathways are interconnected and the cognitive mechanisms are very complex. Consequently, there is no consensus for one human vision model.

We believe that the future of image understanding will advance through the development of human vision models which better take into account the hierarchy of visual image processing stages from the preattentive stage to the interpretation stage. With such a model, we could bridge the gap between low-level descriptors and high-level interpretation. With a better knowledge of the interpretation stage of the human vision system we could analyze images at the semantic level in a way that matches human perception.

3. Color Image Applications: Issues, Controversies, Problems

When we speak about color image science, it is fundamental to evoke firstly problems of acquisition and reproduction of color images but also problems of expertise for particular disciplinary fields (meteorologists, climaticians, geographers, historians, etc.). To illustrate the problems of acquisition, we evoke the demosaicking technologies. Next, to illustrate the problems with the display of color images we speak about digital cinema. Lastly, to illustrate the problems of particular expertise we quote the medical applications.

3.1. Color Acquisition Systems

For several years, we have seen the development of single-chip technologies based on the use of color filter arrays (CFAs) [12]. The main problems these technologies have to face are the demosaicking and the denoising of resulting images [13–15]. Numerous solutions have been published on facing these problems. Among the most recent ones, Li proposed in [16] a demosaicking algorithm in the color difference domain based on successive approximations in order to suppress color misregistration and zipper artefacts in the demosaicked images. Chaix de Lavarène et al. proposed in [17] a demosaicking algorithm based on a linear minimization of the mean square error (MSE). Tsai and Song proposed in [18] a demosaicking algorithm based on edge-adaptive filtering and postprocessing schemes in order to reduce aliasing error in red and blue channels by exploiting high-frequency information of the green channel. On the other hand, L. Zhang and D. Zhang proposed in [19] a joint demosaicking-zoomingalgorithm based on the computation of the color difference signals using the high spectral-spatial correlations in the CFA image to suppress artefacts arising from demosaicking as well as zippers and rings arising from zooming. Likewise, Chung and Chan proposed in [20] a joint demosaicking-zoomingalgorithm based on the interpolation of edge information extracted from raw sensor data in order to preserve edge features in output image. Lastly, Wu and Zhang proposed in [21, 22] a temporal color video demosaicking algorithm based on the motion estimation and data fusion in order to reduce color artefacts over the intraframes. In this paper, the authors have considered that the temporal dimension of a color mosaic image sequence could reveal new information on the missing color components due to the mosaic subsampling which is otherwise unavailable in the spatial domain of individual frames. Then, each pixel of the current frame is matched to another in a reference frame via motion analysis, such that the CCD sensor samples different color components of the same object position in the two frames. Next, the resulting interframe estimates of missing color components are fused with suitable intraframe estimates to achieve a more robust color restoration. In [23], Lukac and Plataniotis surveyed in a comprehensive manner demosaicking demosaicked image postprocessing and camera image zooming solutions that utilize data-adaptive and spectral modeling principles to produce camera images with an enhanced visual quality. Demosaickingtechniques have been also studied in regards to other image processing tasks, such as compression task (e.g., see [24]).

3.2. Color in Consumer Imaging Applications

Digital color image processing is increasingly becoming a core technology for future products in consumer imaging. Unlike past solutions where consumer imaging was entirely reliant on traditional photography, increasingly diverse color image sources, including (digitized) photographic media, images from digital still or video cameras, synthetically generated images, and hybrids, are fuelling the consumer imaging pipeline. The diversity on the image capturing and generation side is mirrored by an increasing diversity of the media on which color images are reproduced. Besides being printed on photographic paper, consumer pictures are also reproduced on toner- or inkjet-based systems or viewed on digital displays. The variety of image sources and reproduction media, in combination with diverse illumination and viewing conditions, creates challenges in managing the reproduction of color in a consistent and systematic way. The solution of this problem involves not only the mastering of the photomechanical color reproduction principles, but also the understanding of the intrinsic relations between visual image appearance and quantitative image quality measurements. Much is expected from improved standards that describe the interfaces of various capturing and reproduction devices so they can be combined into better and more reliably working systems.

Free download research paper