The sensor is made up of millions of photosensitive cells arranged in rows and columns.
For example, a 24Mpx sensor usually has an array of 6000 x 4000 photosensitive cells.
We will initially assume a monochrome sensor (which only generates black and white images).
Each cell collects an amount of light, which corresponds to that small area of the scene, and converts it into a proportional numerical value.
The darkest points in the scene (called blacks or shadows) will have low values or levels closer to zero.
The brightest or lightest points in the scene (called lights or whites) will have values close to the full scale.
For example, if the level scale is from 0 to 255, the purest white would be the value 255. There would be nothing brighter in the final image.
For a monochrome sensor, the story would end here.
But most cameras produce color images, so let’s see how they do it.
Color sensors
Although there are different options and technologies, the technique used in most cameras is based on the following:
- A single sensor is used.
As we have seen, the sensor itself only detects levels or quantity of light. - An RGB filter is placed on the sensor.
This optical filter is made up of a mosaic of cells that are distributed following a particular pattern. The most widely used is the Bayer pattern, but for example, Fuji uses its slightly different X-Trans pattern. - Each cell of the mosaic allows only part of the light to pass.
through R cells only let through photons of wavelengths around red
G cells only let through the green light.
B cells only let through the blue light.
In both Bayer and X-Trans filters, greens are more prevalent.
These filters are RGGB (two cells dedicated to green for each cell dedicated to red and blue)
This is so because the human eye (human vision) is most sensitive to the visible spectrum’s central part, the entire range of greens.
The photosensitive material used in the sensors usually performs better in that central area of the spectrum. We could say that the green color better represents the brightness of the scene.
When the light coming from the scene passes through the RGB filter, each sensor cell will only receive information from its corresponding component.
For example, a particular cell will have the red level corresponding to that point in the scene. The adjacent cell will have the green level for that other point and so on.
We are dividing the scene into color channels (the red channel, the green channel, and the blue channel)
If our sensor were 24Mpx, we would have:
- A 6MP black and white image corresponding to the blue channel
- A 6MP black and white image corresponding to red tones
- A 12 Mpx (2 x 6 Mpx) black and white image corresponding to greens
These three images are superimposed but not overlapping.
The information we get from the sensor is simply a matrix of levels, without distinction of any kind, as if it were a monochrome sensor.
RAW image information
That matrix of levels or digital values (numbers) that come out of the sensor is known as RAW values.
If we are taking the photo in RAW format, the camera’s processor will pack that information, as is, in the
A RAW form would add some additional information such as camera configuration parameters and save it to the memory card.
The RAW file is not an image file; it is a container with the sensor’s values without processing.
From that RAW file and knowing the RGB mosaic pattern that the camera uses, we could easily extract each color channel separately.
We would have three images, three versions of the same scene, each one according to one of the color components’ points of view.
But we do not want that; what we want is a single image with the maximum resolution of the sensor, in which each point has the exact information of its color.
That is, each pixel has its totally defined tonality, its three RGB components: for example R = 20, G = 80, B = 125
This is what is achieved with the color interpolation or color interpolation algorithms.
Color interpolation (demosaicing)
Its name usually knows the process in English: demosaicing.
Each point in the image only has information on one color, and we have to estimate the two missing colors.
For example, if we are in a cell that corresponds to the green component, we could look at the surrounding blue cells’ values to estimate the blue part. And the same for the red piece.
A simple way would be to use some linear averaging of the adjacent cells of the color we are trying to figure out.
For example, to estimate the blue component of a pixel, we can look at the adjacent blue cells’ values, calculate the sum of their levels, and divide by the number of cells to calculate the mean.
There is no single possible result, and there is no direct correspondence between the three color channels and the final image.
In a way, we are ‘inventing’ the color of the image.
From a RAW file, we can obtain many different versions of the final image depending on the algorithm used for interpolation.
The problem with linear interpolation (linear averaging) is that at the edges of objects in the scene, in sharp color transitions in the image, color artifacts and small repetitive patterns appear (known as zippers, zippers)
Modern algorithms are quite complex and use techniques to minimize such unwanted effects. Some are optimized to maintain edge detail and minimize zipper patterns, others to preserve highlights, others to protect colors.
There are many, many algorithms and variants. In any case, for a photographer, the demosaicing process is usually totally transparent, and it is complicated to notice these adverse effects with current sensors (high resolution) and with the standard algorithms.
Where / when is the color interpolation done?
If we take the RAW format on the camera, the color interpolation will be done later by the development program we use: Lightroom, Capture One, Luminar, Dark Table …
In some programs, the process is transparent to the user; in others, we can choose between different algorithms to best suit the newborn photo.
The development program must have each camera’s information and the structure and specific characteristics of the RAW information generated by a said camera in its database.
Let’s take the photo in JPEG directly on the camera. The camera’s processor is internally responsible for the chromatic interpolation and all the processes necessary to achieve the final image.