TRIPS Disagreement Results

  • Influence on Listener Kappa 0.59, PA 0.80
  • Influence on Speaker Kappa 0.27, PA 0.75
  • Abandoned Kappa Insignificant
  • Agreement Kappa .52, PA .77
  • Answer Kappa .59, PA .91
  • Conventional almost perfect
  • Exclamation perfect
  • Performative Kappa Insignificant
  • Info-level Kappa 0.55, PA 0.87
  • Other Forward Function Kappa Insignificant
  • Response-to Kappa 0.67, PA 0.78
  • Self-talk perfect
  • Statement Kappa 0.32, PA 0.70
  • Understanding Kappa 0.4, PA 0.77
  • Unintelligible Kappa Insignificant
  • Info-request Kappa 0.62, PA 0.88

    Comments on TRIPS Dialog 971202.1238

    General Comments

    email mcore@cs.rochester.edu to add your comments.

    comments by annotator C: not sure whether about OK and alright in acceptances (like utt59): should they be assert? "done" seems to be, "fine" not sure. where to draw the line?"

    editor's response: OK and alright are not asserts. They don't say anything about the world like "done" does. I'd group "fine" with "OK" and "alright"; they all describe attitudes toward the previous utterance(s).

    comments by mcore: First, I plan on changing the dat warnings to remind people to fill in info-level and to not expect directives to have answers if they are not understood.

    In my annotations as well as those of others I noticed a problem with utterances such as utt7-utt9. There seems to be a backwards function from utt9 to utt8 Perhaps we need an understanding tag such as "repeat-misheard" (utterance).

      utt7 U: CLEAR [CLEAR should be: WHERE ARE] THE PEOPLE
      utt8 S: I'm afraid I didn't understand you
      utt9 U: where are the people
      
    Another problem is capturing the propose-revise-accept sequence often seen in task-oriented dialogs. Perhaps we need a "revise" tag in the agreement dimension to capture situations such as utt21 below. We also need to make sure to keep a firm distinction between higher-level discourse structure and DAMSL tags so that we are not doubly coding features.
      utt18+19 U: How long will that take
      utt20 S: It will take 6 hours and 54 minutes
      utt21 U: what if we went along the coast instead
      
    Labeling the dialog composed of utt40-utt43 is tricky. My approach was to label utt40 as just an assert and leave it to the higher-level discourse people to identify a referent for "that" and establish the accept relation themselves. I think the same needs to be done for utt43. Finding the highlighted actions and labeling them as possibly rejected (maybe's) seems beyond the capabilities of DAMSL. I think there are three equally valid approaches to utt42:

  • simply label it and utt43 as asserts.

  • label it info-request and assert (an indirect question, "what changes when the bridge is out?"). Utt43 is then an answer.

  • label it action-directive and assert ("the bridge is out please change the plan accordingly"). Now one might label utt43 as rejecting the action-directive ("these actions may need changing, you do it").
      utt38 u: and take the people there to delta
      utt39 s: alright
      utt40 u: that looks good
      utt41 s: no problem
      utt42 u: Delta bridge is out
      utt43 s: the highlighted actions may need to be revised
      
    It seems like TRIPS has bags of supposedly equalivalent utterances ("done","your wish is my command", "okay") and ("I didn't understand", "Pardon me?") which I labeled somewhat differently. I labeled "your wish is my command" as a statment but it is idiomatic and I probably shouldn't have. "done" I labeled as statement but not accept or acknowledge. I think it is truly different than "okay" or "alright" which can happen before the action is performed. I also think the acceptance and acknowledgment in "done" is implicit. I labeled "Pardon me?" as an information request since it means "please repeat yourself". One could say the same thing about "I didn't understand" but I didn't feel that way as I could imagine the TRIPS system happily never understanding what you last said. Treating "Pardon me?" as a question sets off one of the dat warnings as it expects an assert as an answer. It would be nice to fix that.

    Mystery of Utterances 63 through 69

      T54      utt63    U: WHERE ARE THE PEOPLE
      T55      utt64    S: There are 2 groups-of-people at Delta
      T56      utt65    U: WHERE ARE ALL THE PEOPLE
      T57      utt66    S: There are 6 groups-of-people at Delta and 1 at South Delta
      T58      utt67    U: USE THE HELICOPTER
      |        utt68    U: TO GET THE PEOPLE FROM SOUTH_DELTA TO DELTA
      T59      utt69    S: There are no people at South Delta
      T60      utt70    U: USE THE HELICOPTER TO GET THE PEOPLE AT SOUTH_DELTA TO DELTA
      T61      utt71    S: Your wish is my command
      

    My best guess: definite reference for THE PEOPLE in utt63 is wrong. It gets mapped to the lastest truckload of people which is 2 groups. ALL clarifies things and we get all 6. Utt63 is interpreted as "where are the people we just dropped off?". Utt67 and utt68 are interpreted as referrering to something like the people orginally from south_delta (none). The generator unfortunately chooses to express this by saying "at south delta". PEOPLE AT SOUTH_DELTA is resolved correctly to the people currently there.

    Comments on Individual Utterances

      1     S: Welcome to trips 97 version 3 point 2. 
      2       I'm ready to start. 
      
      M - open option - let's start
      C - task-man or comm-man? manual gives TM  precedence
      
      3     U: HELLO 
      4     S: Hi 
      
      M - no backward function for greetings
      
      5     U: SHOW_ME A MAP OF PACIFICA 
      
      M - showing map is answer
      C - initially marked as IR, revised later
      
      6     S: Ok 
      7     U: CLEAR [CLEAR should be: WHERE ARE] THE PEOPLE 
      8     S: I'm afraid I didn't understand you. 
      
      D - implicitly asking the speaker to repeat?
      M - indirect question?
      
      9     U: WHERE ARE THE PEOPLE 
      10     S: There are 2 groups-of-people at Exodus, 2 at Calypso, 2 at Barnacle 
              and 1 at Abyss 
      
      A - I don't like coding all answers as acknowledging, but I feel I have no 
          choice --- you say all accepts also acknowledge
      
      11     U: WHERE ARE THE TRANSPORTS 
      12     S: There are 2 trucks and 1 helicopter at Delta 
      
      D - not open option  because the speaker is not suggesting potential
            actions.
      
      13     U: USE A TRUCK [missed: TO] GET THE PEOPLE FROM CALYPSO 
      14       TO DELTA 
      
      E - In Verbmobil, we perfer functional prosodic segmentation instead
       of auditory prosodic features to get rid of such boundaries
      
      15     S: Ok 
      
      B - ok, yes here often got an agreement tag because system seems to 
          confirm not  only message understanding (like in maptask or verbmobil) but also 
          performance of the action 
      
      16     U: IT [IT mistakenly inserted] HOW_LONG WILL A [A should be THAT] TAKE 
      17     S: I'm afraid I didn't understand you. 
      
      D - Not clear if it is a hold.  The first question in the agreement 
            decision tree is hard to answer.
      
      18     U: HOW_LONG 
      19       WILL THAT TAKE 
      20     S: It will take 6 hours and 54 minutes 
      21     U: WHAT IF WE WENT ALONG [missed: THE] COAST INSTEAD 
      
      D - If this were two people planning together, I would say open option.  
           But  here it looks like the user is just trying to get more information.
      
      C - no influence-on-S even if plural subject, just an idiom
      
      22     S: I'm sorry, but that input didn't make sense to me. 
      
      A - unclear whether S sees utt19 as open-option, so somewhat strange to code as 
          "hold" --- I suppose coding from human perspective?
      
      23     U: WHAT IF WE WENT ALONG THE COAST INSTEAD 
      
      M - open-option, unaddressed by 24, dat warning comes up, yuck
      
      24     S: That option would take 8 hours and 52 minutes 
      
      A - unclear if S sees utt21 as open-option or straight info-request
      
      25     U: I [I mistakenly inserted] FORGET IT 
      
      C - assert, not influence-on-L,  because of "I"
      
      26     S: Ok 
      
      C - no influence-on-S because of  antecedent
      
      27     U: USE THE OTHER TRUCK [missed: TO] GET THE PEOPLE AT EXODUS TO 
              DELTA 
      
      D - Was the 6 hours 54 minutes implicitly accepted when the 8 hours 
               52 minutes was rejected?
      
      28     S: Ok 
      29     U: LET'S USE A HELICOPTER INSTEAD 
      
      C - no "influence-on-S" because actions are of "s"
      
      30     S: Alright 
      31     U: NOW SEND TRUCK ONE 
      32       TO BARNACLE 
      33     S: I'll get right on it 
      34     U: PICK UP THE PEOPLE THERE 
      35     S: Ok 
      36     U: GO ON TO ABYSS 
      37     S: Your wish is my command 
      
      D - This and "I'll get right on it"  are really just  like "ok" coming 
            from this machine, but I tagged them differently.
      
      38     U: AND TAKE THE PEOPLE THERE TO DELTA 
      39     S: Alright 
      40     U: THAT LOOKS GOOD 
      41     S: No problem 
      
      A - not sure what system means; take as confirming plan works
      C - not quite sure what utt41 does
      D - acknowledgements that are not accepts are usually communication 
            management.
      
      42     U: DELTA_BRIDGE IS OUT 
      
      F - i need to know more about task to label this correctly
      
      43     S: The highlighted actions may need to be revised. 
      
      F - i need to know more about task to label this correctly
      C - could it be TM? could it be act-Dir and/or Offer? would need info-rel as well
      
      44       Ok 
      
      F - can't label this without knowing more about the task
      D - This "ok" doesn't  seem to be well-formed.  I'm assuming that it
            is a signal understanding that would have sounded better if it came before 
            utterance 44.
      
      B - s maintains the turn by uttering ok
      C - not sure about function of this OK
      A - don't understand why S says OK here.  Ready to listen again?
      M - accept knowledge of 42
      
      45     U: SEND_A [A mistakenly inserted] TRUCK TWO 
      46       ALONG THE COAST INSTEAD 
      47     S: Sorry, I don't know how to do that. 
      
      C - interpreted as "I can't comply" instead of "I don't understand"
      
      48     U: SEND TRUCK TWO 
      49       ALONG THE COAST INSTEAD 
      50     S: Done 
      
      C - uncertain about  commit because action already done
      
      51     U: HAVE THE OTHER TRUCK STOP AT SOUTH_DELTA INSTEAD 
      52     S: Alright 
      53     U: UNLOAD A [A should be THE] PEOPLE 
      54     S: Pardon me? 
      
      D - responses to directives usually have an agreement feature
      C - comment=performative because equivalent to "sorry" (???)
      
      55     U: UNLOAD 
      56       THE PEOPLE 
      57     S: Fine 
      
      C - comment=different from "ok, alright" as statement?
      
      58     U: USE THE HELICOPTER [missed TO] GET THE PEOPLE FROM SOUTH_DELTA TO DELTA 
      59     S: Alright 
      60     U: WHERE ARE THE PEOPLE 
      61       LEFT [LEFT mistakenly inserted] 
      62     S: I don't understand your reference to the people left 
      
      D - I thought I saw somewhere in the manual that "I don't know"  is 
           a reject because it's like saying I can't open the door because 
           my arm is broken.
      
      63     U: WHERE ARE THE PEOPLE 
      64     S: There are 2 groups-of-people at Delta 
      65     U: WHERE ARE ALL THE PEOPLE 
      66     S: There are 6 groups-of-people at Delta and 1 at South Delta 
      67     U: USE THE HELICOPTER 
      68       TO GET THE PEOPLE FROM SOUTH_DELTA TO DELTA 
      69     S: There are no people at South Delta 
      
      C - don't understand utt69 see 66
      A - could be hold, not reject; submissive systems don't ever reject plans 
            but point up flaws.
      M - misunderstanding, but system doesn't know. can we mark this separately
            from signal non-understanding
      
      70     U: USE THE HELICOPTER TO GET THE PEOPLE AT SOUTH_DELTA TO DELTA 
      71     S: Your wish is my command 
      72     U: [user selects simulate plan from a menu] 
      
      F - how to label this?
      D - no verbal action/ no tags
      C - should it be "commit"? but actions are of the system
      
      73     S: Simulation succeeded in 94 percent of 250 runs. 
      
      C - comment=should it be accept? no, don't think so (see discussion about 
            "done" in utt50)
      
      74       7 percent of the plan failure points were a move action with 
              truck two resulting in a breakdown. 
      

    Last change: 05 May 98 by mcore

    Click here to send in comments or questions.

    ^^
    Back to the Dialog Annotation Page