What is color interpolation (demosaicing)?

The sensor is made up of millions of photosensitive cells arranged in rows and columns.

For example, a 24Mpx sensor usually has an array of 6000 x 4000 photosensitive cells.

We will initially assume that it is a monochrome sensor (which only generates black and white images).

Each cell collects an amount of light, which corresponds to that small area of ​​the scene, and converts it into a proportional numerical value.

The darkest points in the scene (called blacks or shadows) will have low values ​​or levels, closer to zero.

The brightest or lightest points in the scene (called lights or whites) will have values ​​close to the full scale.

For example, if the level scale is from 0 to 255, the purest white would be the value 255. There would be nothing brighter in the final image.

For a monochrome sensor the story would end here …

But most cameras produce color images , so let’s see how they do it.

Color sensors

Although there are different options and technologies, the technique used in most cameras is based on the following:

  • single sensor is used
    As we have seen, the sensor itself only detects levels or quantity of light
  • An RGB filter is placed on the sensor.
    This optical filter is made up of a mosaic of cells that are distributed following a certain pattern. The most widely used is the Bayer pattern, but for example Fuji uses its slightly different X-Trans pattern.
  • Each cell of the mosaic allows only part of the light to pass
    through R cells only let through photons of wavelengths around red
    G cells only let through green light
    B cells only let through blue light

In both Bayer and X-Trans filters, greens are more prevalent .

Actually these filters are RGGB (two cells dedicated to green for each cell dedicated to red and blue)

This is so because the human eye (human vision) is most sensitive to the central part of the visible spectrum, the entire range of greens.

And also the photosensitive material used in the sensors usually has a better performance in that central area of ​​the spectrum. We could say that the green color better represents the brightness of the scene.

 

When the light coming from the scene passes through the RGB filter, each sensor cell will only receive information from the component that has corresponded to it.

For example, a certain cell will have the level of red that corresponds to that point in the scene. The adjacent cell will have the green level for that other point… and so on.

We are dividing the scene into color channels (the red channel, the green channel and the blue channel)

If our sensor was 24Mpx we will have:

  • A 6MP black and white image corresponding to the blue channel
  • A 6MP black and white image corresponding to red tones
  • A 12 Mpx (2 x 6 Mpx) black and white image corresponding to greens

These three images are superimposed, but not overlapping .

In fact, the information we get from the sensor is simply a matrix of levels, without distinction of any kind, as if it were a monochrome sensor.

RAW image information

That matrix of levels or digital values ​​(numbers) that come out of the sensor are what are known as RAW values.

If we are taking the photo in RAW format, the camera’s processor would simply pack that information, as is, in the RAW format used by the manufacturer, it would add some additional information such as camera configuration parameters, etc. and save it to the memory card.

The RAW file is not an image file as such, it is a container with the values ​​collected from the sensor, without processing.

From that RAW file and knowing the RGB mosaic pattern that the camera uses, we could easily extract each color channel separately.

We would have three images, three versions of the same scene, each one according to the point of view of one of the color components.

But we do not want that, what we want is a single image with the maximum resolution of the sensor, in which each point has the exact information of its color.

That is, each pixel has its totally defined tonality, its three RGB components: for example R = 20, G = 80, B = 125

This is what is achieved with the color interpolation or color interpolation algorithms.

Color interpolation (demosaicing)

The process is usually known by its name in English: demosaicing

Each point in the image only has information on one color, and we have to estimate the two missing colors .

For example, if we are in a cell that corresponds to the green component, we could look at the values ​​of the surrounding blue cells to make an estimate of the blue component. And the same for the red component.

A simple way would be to use some kind of linear averaging of the adjacent cells of the color we are trying to figure out.

For example, to estimate the blue component of a pixel, we can look at the values ​​of the adjacent ‘blue’ cells, calculate the sum of their levels, and divide by the number of cells to calculate the mean.

Think that there is no single possible result, there is no direct correspondence between the three color channels and the final image.

In a way we are ‘inventing’ the color of the image.

From a RAW file we can obtain many different versions of the final image depending on the algorithm used for interpolation.

The problem with linear interpolation (linear averaging) is that at the edges of objects in the scene, in sharp color transitions in the image, color artifacts and small repetitive patterns appear (known as zippers, zippers)

Modern algorithms are quite complex and use techniques to minimize such unwanted effects. Some are optimized to maintain edge detail and minimize zipper patterns, others to preserve highlights, others to preserve colors …

There are many, many algorithms and variants. In any case, for a photographer the demosaicing process is usually totally transparent and it is very difficult to notice these negative effects with current sensors (high resolution) and with the usual algorithms.

Where / when is the color interpolation done?

If we take the photo in RAW format on the camera, the color interpolation will be done later by the development program that we use : Lightroom, Capture One, Luminar, Dark Table …

In some programs the process is transparent to the user, in others we can choose between different algorithms to see which one best suits the photo we are developing.

The development program must have in its database the information of each camera and the structure and specific characteristics of the RAW information generated by said camera.

 

If we take the photo in JPEG directly on the camera , it is the camera’s processor that is internally responsible for the chromatic interpolation and all the processes necessary to achieve the final image.