In this section we develop the information theoretic concepts needed for scale and spatial expansion of the information in images.
Shannon defines the information obtained from the observation of an event A with probability p(A) as:
In a continuous setting we can regard
as
an information infinitesimal. p(x) is a probability distribution
function of a random variable x.
The Kullback contrast between two distributions is defined as:
It can be rewritten as:
where
.
This captures our intuitive notion of a contrast. It is positive, and
acts as an asymmetric weighted average of the difference in information
between the two distributions. It is zero if and only if P=Q.
We want to study how information is distributed in different spatial scales. One way to do that is to let the information be ``diffused'' before we observe it. We model this behavior with a diffusion equation on a distribution p(r,x), where r can be interpreted as a resolution length.
with initial condition
and
Neumann boundary condition
.
is
the boundary points of our set of image points I, and
is a
normal vector to
the boundary.
The choice of
boundary condition here is crucial. The information theoretic concepts on
lattices have been developed on doubly infinite n-dimensional sequences.
We want to use them on images, which are ``small cutouts'' of reality (that is
only defined on a bounded, finite set I of x values).
Using for instance a Dirichlet boundary condition would introduce
artifacts not in the original distribution.
An interpretation in our image application of this is that a Neumann boundary condition corresponds to an assumption of continuity over the image boundary, that is the world looks much the same on both sides. It acts also to preserve the total probability within the bounded set of support for our image.
The
diffusion equation is conservative in that if
then
and the first moments of p(x) and p(r,x) are
the same.
A solution is:
where J is the gradient of p (
), and H is the Hessian.
Identifying the above as a Taylor expansion of:
gives us the result as a convolution.
In the experiment section we study information decomposition ID(2r) as the
successive contrast
K[p(r,x),p(2r,x)] between lengths of resolution r and 2r.
That yields a scale space expansion for
,
,
...
.
There are many possible choices of probability distributions over images suitable for the information decomposition. We want our information measure to relate to the interpretation of the image, that is p(x) should reflect some property we (intuitively) think is relevant. To motivate a simple choice recall that the Shannon definition assigns information according to the ``sharpness'' of the distribution that an event occurs, that is, a well localized (sharp) distribution is assigned a high information value. Similarly we get more information when looking at a sharp image than a blurred one.
So a simple choice is just to normalize the
intensity image,
where
is a two-dimensional random variable and Intensity(x)
is the intensity of a grey scale image.
An information decomposition defined on the above p(x) draws attention to local contrast. It turns out to give results that remarkably well correspond to our intuitive idea of the scales and spatial coordinates at which the information in the image is located.
Among other possible measures we try a line-oriented one, ``lininess'', in Section 6. This measure is obtained by convolving the image with Gabor patches used as line finders of different scales and orientations, similar to those of directionally sensitive simple cells in human visual cortex [7]. This helps ``draw attention'' to objects, rather than texture when for instance we have a high contrast random noise like texture in an image.