In this chapter Ballard covers basic concepts in
unsupervised learning. It starts
principal component analysis,
firstly outlined in Chapter 4, now observing that this interesting
processing can be performed inherently, and without prohibitive computational demands, by some neural networks.
This is accomplished by using a two-layer neural network using linear units, with evolution governed in such a
way as to find the eigenvector associated to the largest eigenvalue. However, the reader is refereed to the
literature for an extension allowing to find more eigenvalues in descending order [Sanger].
The next section introduces and characterizes the concept of competitive learning, followed by a section on topological
constraints, which is finely motivated and illustrated with respect to the
travelling salesman problem.
The issue of supervised competitive learning is discussed in the subsequent section. Here, the motivation
is to allow supervision in the initial definition of the space partitions in order to secure a more rational
and effective classification. The biologically very relevant issue
of multimodal data is covered in the next
chapter in terms of a technique based on gradient descent over an energy function defined in terms of joint
density functions describing disagreement between modalities. This algorithm is outlined in a practical way
and illustrated for the case of simultaneous visual and acoustic signals. The last section in this chapter
addresses independent components. The beginning of this section provides a good example of a situation which
tends to recur from time-to-time, but fortunately not too frequently, in Ballard's book, especially in the
last sections in some chapters (see Sections 4.6, 5.3.2 and 7.5). As a matter of fact, the first paragraph
in Section 9.7 reads as notes for somebody who is already quite familiar with the covered topic. If not,
what is a "cause" or "source" s? In which sense is a cause "hidden" or "independent"? Why should causes
be experienced (by who or what?) as data (what does not qualify as data?) because of the transformation
x = As? What is x? What is matrix A? Why should a compact code be one that recovers the original causes?
It is rather unlikely that such an introduction will make any sense to the novice reader.