A few times a minute, the system takes a general view of the campus.
This is the image that is displayed as the current view
(though the web image is only updated every two minutes).
The overall brightness of this image, along with recent history, is used
to decide whether it is daytime or nighttime.
A few times an hour, the system moves the camera to look at the sky
(and also the ground). During the day, the sky image is analyzed to
determine the current degree of cloudiness.
The ground image is currently not analyzed; possible uses include determining
snow cover and season.
The system also saves general views taken near dusk and dawn.
Sometimes these make interesting viewing.
The day-night determination is not a simple threshold computation. It was designed to be robust against disturbances such as lightning or car headlights hitting it at night, and against brief obscuration (e.g. by a person blocking the lens) during the day. It is also adaptive and self-bootstrapping. Unlike a simple threshold system, the performance is not sensitive to the particular camera used or to the environment that it is viewing. This is accomplished by creating and maintaining a statistical model of the distribution of observed brightness. The system locks onto bimodality in the statistics and generates models for day and night brightness distributions. The system also maintains various time averages of the brightness. These are used in evaluating expectations of the transition from day to night and vice-versa. These expectations are employed to prevent oscillations in the day/night determination near transition points, as could occur with a threshold system if the sun went in and out of a cloud deck at sunrise or sunset.
Near transition times, the system may report a "twilight" condition. It can also enter a "confused" mode. This would occur, for instance, if it were placed in a room where lights were turned on and off sporadically, with no daily rythm. The system's response to being confused is to refrain from making a day-night decision for a period that increases exponentially with repeated confusion (up to a limit of about 24 hours). If "normal" conditions resume, the system gradually recovers from its confused state. While confused, the system will report "bright-as-day" or "dark-as-night" rather than "it is day" or "it is night".
The system adapts its statistical models by accumulating information over a nightfall to nightfall period. This daily accumulation is combined with the long term models during an "REM-sleep" operation that is invoked each day after nightfall. The batch processing aspect avoid instabilities and poor conditioning that would result if the updates were done after each observation, since it is day or night for thousands of consecutive observations.
The system's day-night perception is based to some degree on
biological system considerations - an animal that is diurnal or nocturnal,
that must deal with a variable environment with potential
disruptions, and for which engaging in nighttime activity during the day
or vice versa (i.e. making a mistake) could have a high price.
The determination of the cloud coverage in daytime images also employs the ongoing collection of statistical information, in this case about the color of pixels in the sky. Analysis of the distribution reveals that most of the variation in sky color occurs along the normalized (blue - red) dimension, so a 1-dimensional analysis suffices. Again, a bimodality in the distribution forms the basis for statistical models for clear and cloudy portions of the sky. Unlike the case for the day/night analysis, a single image supplies many data points; however, the distribution of points over the classes is extremely non stationary. An overcast sky supplies no examples of clear pixels, and (in Rochester) there can be two contiguous weeks of overcast skies. Consequently, the adaptation of the statistical data must be done with care. Currently this is handled by ensuring that the daily data accumulation is bimodal before attempting to combine it with the permanant models. The bimodality itself is more subtle than in the day-night data. Frequently, the presence of two distributions is reflected only in the skewing or broadening of a peak, rather than in clearly defined local maxima. Currently, Gaussian mixture models in conjunction with the EM algorithm are employed to pull out the structure. (They are also used for the day-night data, but are not really needed there.)
There are a number of meteorological complications. If the sky data are considered as a whole, models for clear and cloudy pixels can be easily extracted. In individual images however, especially ones taken near dusk or dawn, a number of dramatic deviations from the norm occur. Brilliant reds and golds from sunset effects are the most obvious examples, however these are usually easy to recognize. The most problematic effect is what we term the "blue shift" phenomenon. Near dusk (and dawn) a cloudy sky becomes increasingly blue, to the point where it is sometimes as blue as clear sky at midday. (Even the human visual system, which is amazingly good at normalizing out chromatic effects, notices this). The effect can be detected and compensated for by considering other sources of information. The absolute illumination is a clue, but since we generally use cameras with automatic exposure turned on, this information is not readily available. Bimodality is a clue. If there is mixed sky and clouds, the blue shift can be computed from the location of the cloud peak. In unimodal images, clear sky can be often distinguished from overcast by the vertical gradient. Under clear conditions, the horizon is always ligher than higher elevations (due to the geometry of the paths through the atmosphere and the physics of Raleigh scattering). Uniform overcast is usually darker near the horizon, hence a darker-towards-the-horizon vertical gradient is a near-certain indicator of an overcast sky.
After all these convolutions, the pixels from a sky image are
classified as clear, cloud, or hazy (in between). There is also
a "red-sky" class for handling sunset effects.
The overall cloudiness condition is determined to be one of
clear, mostly clear, partly cloudy, mostly cloudy, and cloudy, based
on the proportion of pixels in the various classes.
(The system will also comment on nice sunrises or sunsets, but
I have never seen it do so, since it generally avoids the sun by
looking to the northeast.)
The system counts the number of times that the various classes
occur over the course of a day, which permits the
"cloudy-all-day" etc. statements sometimes reported under events.
Future enhancements to the system may include dection of snowing or raining
conditions, detection of snow on the ground, and even estimation of
windy conditions by watching trees move (though first we need to
be able to detect trees, if there are any)