Graphics, Imaging, and Games Lab

The Residual

The "residual" from filtering, as used in this post, is the subtraction of the filtered image from the original image. Larger magnitudes in the residual indicate areas of higher-frequency colour variation or outliers in the source image.

Consider the following pair of images below: The first image is the source image, while the second is the residual. A linear transformation was applied to the colours of the residual to map RGB channel values from their original range of -110.25 to 164.7344 to the range of 0-255, where values of zero in the residual were mapped to 128. In other words, areas where the image and its smoothed version do not differ significantly are shown as middle grey in the linear transformation of the residual.

All results in this post were obtained with a filter "gamma" parameter value of zero.

Source Image

Residual obtained at a filtering mask size of 64

As the residual contains a large range of colour channel values, but extreme values occur very infrequently, it makes sense to apply a nonlinear transformation to the residual before using the residual in any subsequent process. The recolouration results shown later in this post were generated with a transformed residual obtained by taking the base-2 logarithm of the residual (as shown in the image below). (To be exact, if x is a value in the residual, then the transformed value of x is log2(x), 0, or -log2(-x) if x > 1, |x| < 1, or x < -1, respectively.) I also tried a similar transformation which used the square root function, but found that the compressed residual obtained using the base-2 logarithm had a slightly nicer appearance.

Residual obtained at a filtering mask size of 64, transformed by taking the base-2 logarithm

At a larger mask size, I find that the residual itself is actually an interesting result in itself, as it begins to take on the colours of the source image:

Residual obtained at a filtering mask size of 1024, transformed by taking the base-2 logarithm

Another method for attenuating the extreme values in the residual is to filter the image twice and take the residual as the difference between the final image and the first filtered image. In addition to narrowing the range of values in the residual, this procedure decreases the local contrast within regions of the residual and flattens texture.

Residual obtained from subtracting the first and second filtering passes, both at a mask size of 1024, transformed by taking the base-2 logarithm

Texture Enhancement

As the residual isolates the texture and noise in an image, adding the residual to the original image results in texture enhancement and sharpening. The amount of texture enhancement increases with the mask size used to calculate the residual. Consider the two images below, produced by adding residuals obtained using different filtering mask sizes (for a single filtering pass only) to the original image:

Texture enhancement by adding the residual from filtering with a mask size of 64 pixels

Texture enhancement by adding the residual from filtering with a mask size of 1024 pixels

Note that both of the above images were produced without any nonlinear or linear compression of the values in the residual. When I nonlinearly compressed the residual before adding it to the image, I obtained essentially no texture enhancement. Unfortunately, using the uncompressed residuals causes a significant amount of clipping (RGB channel values that fall outside of the range 0-255 and must be clipped to the range of 0-255). Clipping accounts for the loss of detail in the sap droplets on the pinecones, for example.

While not shown above, I also tried a two-pass filtering approach, calculating the residual as the before-after difference of the second filtering pass. The amount of texture enhancement decreased when the first-pass filtering mask size increased, but on the other hand, small features in the image were less affected by clipping.

Using the Residual for Image Recolouration

The results shown in this section were made using the same procedure for recolouring an image described at http://gigl.scs.carleton.ca/node/538, except that weights for averaging the original and new colours were computed by scaling the magnitude of the residual (which was transformed by the base-2 logarithm), rather than counts of mask inclusions, to values between 0 and 1.

Preferential Recolouration of Non-Outlier Pixels

The following two images show the effect of assigning pixels with larger values in the residual a higher weight relative to the new colour:

Recolouration of non-outliers using a residual generated with a mask size of 64

Recolouration of non-outliers using a residual generated with a mask size of 1024

When the residual is produced using a larger filtering mask size, small features in the image are recoloured less strongly.

I also modified the recolouration procedure by using a two-pass filtering method for calculating the residual (the results of which are not shown), and determined that increasing the first-pass filtering mask size results in a loss of texture in the recoloured areas of the image, because the residual becomes more flattened.

Preferential Recolouration of Outlier Pixels

As when I used mask inclusion counts as weights for recolouration, I can invert the recolouration. To generate the images below, I assigned a higher weight to the new colour for pixels with high corresponding values in the magnitude of the residual. This modification causes outlier pixels to be recoloured as opposed to non-outlier pixels.

Recolouration of outliers using a residual generated with a mask size of 64

Recolouration of outliers using a residual generated with a mask size of 1024

In contrast with the results of recolouring non-outliers, I found that, at larger mask sizes, it was actually desirable to use a two-pass method for calculating the residual, with a large first-pass mask size. Smoothing the residual in this fashion shrank the strongly-recoloured areas in the image. However, the overall effect was more or less equivalent to using smaller first-pass and second-pass mask sizes. In conclusion, therefore, I prefer recolouring outlier pixels with a residual obtained by a single filtering pass at a small mask size.

Comparison of Recolouration Methods

When recolouring by averaging the image with a new colour and using the residual as a weighting factor, I find that recolouring outlier pixels leads to a more pleasing image (glowing edges) than recolouring non-outlier pixels.

Regarding the results in the previous post (http://gigl.scs.carleton.ca/node/538), I find that recolouring outliers using counts of mask inclusions is nicer than recolouring outliers using the residual. However, I think that recolouring non-outliers using the residual is nicer than recolouring non-outliers using counts of mask inclusions as weightings.

(Image Source: Bernard Llanos)