In most cases, there were 17 annotators except for the Agreement tag and the Response To label. In the case of the Agreement tag, one annotator did not have time to finish and skipped this tag. Most annotators forgot to mark what their backward functions responded to and only six annotators recorded this information. Other Backward Function and Other Forward Function were each used only once over the 17 annotators and 35 utterances so they were not included in these statistics. Siegel and Castellan in their book, "Nonparametric Statistics for the Behavioral Sciences" show how to test the significance of kappa scores, to see whether a kappa score was a result of chance or reflects the agreement among the annotators. Siegel and Castellan assume kappas are normally distributed. One tailed significance levels are used here. The significance is calculated through calculating z and looking up the significance in a table. The equations below show how z is calculated. Understanding had a significance level above .1 showing that the kappa was not significant. Looking at the results of the understanding labeling shows a high degree of disagreement making the estimate of kappa variance high giving the non significant rating. Response-to had a negative variance (-0.008362); it is unclear what this means. The kappa for Response-to is included below under the assumption that it is significant. The rest of the results have positive variances and significant kappas; their significance levels are included in the results section.
A problem faced by Understanding and Agreement is that these phenomena can be signaled implicitly ("so buy your $300 yellow sofa" implicitly accepts "The cheapest sofa I have costs $300").
The example below shows a common pattern of disagreement. Here, four people considered both 11 and 12 as acceptances while eight people thought the acceptance only consisted of 11. I checked the dialogs by hand to make sure the four people marking utterance 12 as accept also marked utterance 11 as accept.
None Accept 11 4 | 12 | :STEVE] : You are right, 12 12 | 4 | :: we have $1000 to spend.Implicit actions are even more of a problem for Understanding. At most of the change of turns, a few annotators would mark signal understanding/acknowledgment because by not saying "I didn't understand" you implicitly signal understanding. Utterance 10, "Also...Your $550 + my $450 is $1000", is a correct misspeaking of utterance 4, "So we have a total of $900 to spend." It is unclear whether people did not think correct misspeaking was appropriate or like myself did not notice the correction. Utterances 17 and 18 are not correct misspeaking because the speaker corrects themself. Both of these sets of utterances caused disagreements.
The inter-annotator agreement on Response To reflects disagreements about not only what was responded to but whether an utterance has a backward function (understanding, agreement, answer) at all. The latter of these disagreements will also be reflected in agreement on understanding, agreement, and answer. There were 19 utterances where annotators disagreed on Response To; five of these utterances involved disagreements about the extent of a response. This is understandable as there was no discussion of this labeling in the reference sheet.
Last change: 24 Nov 97 by mcore
Click here to send in comments or questions.