Graphics, Imaging, and Games Lab

Inspiration and Background

Prasun Choudhury and Jack Tumblin used a first-order approximation to the colour of a source pixel when determining the degree of similarity between the source pixel and its neighbours. Their results are described in the following article: The Trilateral Filter for High Contrast Images and Meshes, Eurographics Symposium on Rendering 2003, pages. 1–-11 (edited by Per Christensen and Daniel Cohen-Or).

The overall idea is to describe the neighbourhood of the source pixel with a plane respecting the average colour gradient in the area. We hope this will allow for smoothing which follows gradients in the image more closely than if we simply used averages of pixel values to describe regions.

First Method: Fitted Planes used for Mask Building

Procedure

1. Find an initial sample of pixels which are similar to the source pixel by building a small mask with the normal cumulative range geodesic filtering algorithm.
2. Use MATLAB's backslash operator to perform multiple regression and obtain ordinary least-squares estimates of the coefficients of a plane of best fit for the initial mask. (For a colour image (RGB), there are three planes of best fit, as one plane is needed for each colour channel.) The coefficients of a plane correspond to its constant offset and colour variations with pixel row and column coordinates.
3. Determine the closest pixel to the plane and set it as the starting point for a second mask. Build the second mask the same way as the first, except replace the source pixel's colour channel values with the values of the fitted planes at the locations of the pixels being assessed for inclusion in the mask. In other words, the planes are used for calculating pixel distance values.
   • Note that the starting pixel for the second mask is usually not the source pixel (the starting pixel for the first mask). It is necessary to determine which pixel is closest to the planes because the filter needs to build masks containing the "closest" pixels to the source and, in this case, distances to the source are measured using the planes. For simplicity, I assumed that the closest pixel was one of the pixels in the first mask.
4. Average the colours of the pixels in the second mask to determine the filtered colour of the source pixel. (This is no different from the use of the mask in the normal cumulative range geodesic filtering process.)

Results

Perhaps the most interesting aspect of this approach is that the results are incredibly similar to those produced with the normal cumulative range geodesic filter. A comparison is shown below:

Source Image

Normal cumulative range geodesic filtering (Mask Size of 128, gamma value of zero)

Plane-Fitting Filter (First mask size 24, Second mask size 128)

Experiments with the size of the mask used to fit a plane to the data revealed that the final image is not visibly altered when the 24, 32 or 64-pixel regions are used. However, I needed to use a much larger number of pixels than the minimum number of points required to define a unique plane (three). Due to similarities in colour values and row/column coordinates, multiple pixels often provided equations which were too similar to be considered linearly independent.

The largest differences between the last two images shown above occur at the stronger edges in the source image. It looks like the plane-fitting filter actually sharpens gradients in the image relative to the normal cumulative range geodesic filter, resulting in slightly stronger edges. While this is contrary to what we had hoped, it is actually to be expected. Consider that the first mask made for a pixel on a gradient will tend to grow more on one side of the gradient than the other. This imbalance carries over into the fitted plane, which exaggerates it when the second mask is constructed.

Second Method: Pinning Planes to their Source Pixels

Procedure

This version of the filter uses an algorithm identical to the first method described above except that each plane is forced to pass through the source pixel. (In other words, the plane has a value equal to the source pixel's corresponding colour channel value, at the coordinates of the source pixel.) Therefore, the source pixel is always the starting pixel for the second mask.

I chose to try this method because it is more similar to the plane-fitting procedure described in the above-referenced article on trilateral filtering.

Results

The output images are quite similar to those made with the first plane-fitting method, but seem slightly noisier (mottled), especially along stronger edges. As with the first method, colour gradients appear to be sharpened. It is probable that pinning planes to their corresponding source pixels reduces the filter's ability to remove noise, by better preserving source pixels which happen to be outliers.

 Third Method: Fitted Plane Used Instead of Averaging

Procedure

This filter is actually more similar to the normal cumulative range geodesic filter than to the first two methods described above. It follows the same mask generation procedure as the normal cumulative range geodesic filter. However, it calculates the new colour of a source pixel by fitting planes to the mask and determining the colour channel values of the planes (e.g. Red, Green and Blue) at the location of the source pixel.

Results

This is my favourite of the three filtering methods because it appears to behave as we had hoped: soft gradients remain soft and the filter seems to be very effective at removing noise.  

Unfortunately, the filter runs quite slowly, at a speed of approximately 650 pixels per second when producing the result below. I think there are probably faster filtering methods which can achieve the same result.

Normal cumulative range geodesic filtering (Mask Size of 128, gamma value of zero)  

Plane-Fitting Filter, Third Method (Mask Size of 128, gamma value of zero)

Image source:

Page 2 of Geodesic Image and Video Editing, by Criminisi et al. (ACM Transactions on Graphics, Vol. 29, No. 5, Article 134, Publication date: October 2010.)