Commonsense Reasoning

Commonsense Reasoning ~ Pronoun Disambiguation Problems

Pronoun Disambiguation Problems

Following are two lists: one of sample Pronoun Disambiguation Problems (PDPs), and a list of the challenge PDPs used in the First Winograd Schema Challenge held at IJCAI 2016 .

The passages are taken or modified from existing published works. The sample problems are taken from Carolyn Keene: Nancy Drew: The Quest of the Missing Map, Grosset & Dunlap, 1969, 1942; Jean de Brunhoff: The Story of Babar, Random House, 1933; and Anna Sewell: Black Beauty, J. M. Lupton, 1877. Sentences in WSC-2016 were taken from Laura Ingalls Wilder Little Town on the Prarie, P.G. Woodhouse: Carry on, Jeeves , Sidney Taylor: All of a Kind Family, Esther Forbes: Johnny Tremaine, Ruth Sawyer: The Year of Jubilo, and several other sources.

Sentences have been modified to summarize context and backstory, which are often given over several paragraphs, into a single sentence (PDP 33); to clarify or simplify (PDP 22); or to change gender or number of nouns and pronouns in order to introduce ambiguity (PDP 2). For challenge problems, character names were also changed. Note that a single passage may give rise to multiple Pronoun Disambiguation Problems.

Sample Pronoun Disambiguation Problems

The pronoun to be disambiguated is highlighted in bold. It is repeated again, with a snippet of context, and with several candidate answers, in the line following the passage. The correct answer is given in the next line.

  1. While Nancy and Ellen counted the silverware, Mrs. Smith hastened upstairs. In a few minutes she returned and one look at her stricken face told the girls that the precious map was gone.
    "her stricken face": a) Nancy (b) Ellen (c) Mrs. Smith
    Answer: (c) Mrs. Smith
  2. Sergeant Holmes asked the girls to describe the intruder. Nancy not only provided the policeman with an excellent description of the heavyset thirty-year-old prowler, but drew a rough sketch of his face.
    "rough sketch of his face": (a) Sergeant Holmes (b) the intruder
    Answer: (b) the intruder
  3. After a somewhat lengthy conversation, Nancy convinced the housekeeper that there was no immediate cause for alarm. She did not mention the incident at the Hamilton bus station, knowing it would only add to Mrs. Gruen’s uneasiness.
    "She did not mention": (a) Nancy (b) the housekeeper
    Answer: (a) Nancy
  4. I asked Dave to get me my sweater from the other side of the yacht. While he was gone, I rested my arm on the rail over there and suddenly it gave way.
    "it gave way": (a) sweater (b) yacht (c) arm (d) rail
    Answer: (d) rail
  5. Do you suppose that Peter is responsible for the captain’s illness? Maybe he bribed the cook to put something in his food.
    "he bribed the cook": (a) Peter (b) the captain
    Answer: (a) Peter
  6. Do you suppose that Peter is responsible for the captain’s illness? Maybe he bribed the cook to put something in his food.
    "his food": (a) Peter (b) the captain (c) the cook
    Answer: (b) the captain
  7. Babar wonders how he can get new clothing. Luckily, a very rich old man who has always been fond of little elephants understands right away that he is longing for a fine suit. As he likes to make people happy, he gives him his wallet.
    "he is longing for a fine suit": (a) Babar (b) old man
    Answer: (a) Babar
  8. Babar wonders how he can get new clothing. Luckily, a very rich old man who has always been fond of little elephants understands right away that he is longing for a fine suit. As he likes to make people happy, he gives him his wallet.
    "he likes to make people happy": (a) Babar (b) old man
    Answer: (b) old man
  9. Babar wonders how he can get new clothing. Luckily, a very rich old man who has always been fond of little elephants understands right away that he is longing for a fine suit. As he likes to make people happy, he gives him his wallet.
    "he gives": (a) Babar (b) old man
  10. Babar wonders how he can get new clothing. Luckily, a very rich old man who has always been fond of little elephants understands right away that he is longing for a fine suit. As he likes to make people happy, he gives him his wallet.
    "gives him": (a) Babar (b) old man
    Answer: (a) Babar
  11. Babar wonders how he can get new clothing. Luckily, a very rich old man who has always been fond of little elephants understands right away that he is longing for a fine suit. As he likes to make people happy, he gives him his wallet.
    "his wallet": (a) Babar (b) old man
    Answer: (b) old man
  12. Well satisfied with his purchases and feeling very elegant indeed, Babar goes to the photographer to have his picture taken.
    "his picture": (a) Babar (b) the photographer
    Answer: (a) Babar
  13. Meanwhile, in the forest, the elephants are calling and hunting high and low for Arthur and Celeste, and their mothers are very worried. Fortunately, in flying over the town, an old marabou bird has seen them and come back quickly to tell the news.
    "their mothers": (a) the elephants (b) Arthur and Celeste
    Answer: (b) Arthur and Celeste
  14. Meanwhile, in the forest, the elephants are calling and hunting high and low for Arthur and Celeste, and their mothers are very worried. Fortunately, in flying over the town, an old marabou bird has seen them and come back quickly to tell the news.
    "has seen them": (a) the elephants (b) Arthur and Celeste (c) their mothers
    Answer: (b) Arthur and Celeste
  15. The mothers of Arthur and Celeste have come to the town to fetch them. They are very happy to have them back, but they scold them just the same because they ran away.
    "fetch them": (a) mothers (b) Arthur and Celeste
    Answer: (b) Arthur and Celeste
  16. The mothers of Arthur and Celeste have come to the town to fetch them. They are very happy to have them back, but they scold them just the same because they ran away.
    "They are very happy": (a) mothers (b) Arthur and Celeste
    Answer: (a) mothers
  17. The mothers of Arthur and Celeste have come to the town to fetch them. They are very happy to have them back, but they scold them just the same because they ran away.
    "have them back": (a) mothers (b) Arthur and Celeste
    Answer: (b) Arthur and Celeste
  18. The mothers of Arthur and Celeste have come to the town to fetch them. They are very happy to have them back, but they scold them just the same because they ran away.
    "they scold": (a) mothers (b) Arthur and Celeste
    Answer: (a) mothers
  19. The mothers of Arthur and Celeste have come to the town to fetch them. They are very happy to have them back, but they scold them just the same because they ran away.
    "scold them": (a) mothers (b) Arthur and Celeste
    Answer: (b) Arthur and Celeste
  20. The mothers of Arthur and Celeste have come to the town to fetch them. They are very happy to have them back, but they scold them just the same because they ran away.
    "they ran away": (a) mothers (b) Arthur and Celeste
    Answer: (b) Arthur and Celeste
  21. The first place that I can well remember was a large pleasant meadow with a pond of clear water in it. Some shady trees leaned over it, and rushes and water-lilies grew at the deep end.

    "leaned over it": (a) meadow (b) pond
    Answer: (b) pond
  22. One day Dick was teasing the colts, and did not know that the master was in the next field; but he was there, watching what was going on; over the hedge he jumped in a snap, and catching Dick by the arm, he hit him so hard that he roared with the pain and surprise.
    "he was there": (a) Dick (b) the master
    Answer: (b) the master
  23. One day Dick was teasing the colts, and did not know that the master was in the next field; but he was there, watching what was going on; over the hedge he jumped in a snap, and catching Dick by the arm, he hit him so hard that he roared with the pain and surprise.
    "he jumped ": (a) Dick (b) the master
    Answer: (b) the master
  24. One day Dick was teasing the colts, and did not know that the master was in the next field; but he was there, watching what was going on; over the hedge he jumped in a snap, and catching Dick by the arm, he hit him so hard that he roared with the pain and surprise.
    "he gave": (a) Dick (b) the master
    Answer: (b) the master
  25. One day Dick was teasing the colts, and did not know that the master was in the next field; but he was there, watching what was going on; over the hedge he jumped in a snap, and catching Dick by the arm, he hit him so hard that he roared with the pain and surprise.
    "hit him so hard": (a) Dick (b) the master
    Answer: (a) Dick
  26. One day Dick was teasing the colts, and did not know that the master was in the next field; but he was there, watching what was going on; over the hedge he jumped in a snap, and catching Dick by the arm, he hit him so hard that he roared with the pain and surprise.
    "that he roared": (a) Dick (b) the master
    Answer: (a) Dick
  27. When Mr. Bond, the veterinarian, came to look at the black horse that lay groaning on the grass, he felt him all over, and shook his head; one of his legs was broken.
    "he felt": (a) Mr. Bond (b) the black horse
    Answer: (a) Mr. Bond
  28. When Mr. Bond, the veterinarian, came to look at the black horse that lay groaning on the grass, he felt him all over, and shook his head; one of his legs was broken.
    "felt him all over": (a) Mr. Bond (b) the black horse
    Answer: (b) the black horse
  29. When Mr. Bond, the veterinarian, came to look at the black horse that lay groaning on the grass, he felt him all over, and shook his head; one of his legs was broken.
    "shook his head": (a) Mr. Bond (b) the black horse
    Answer: (a) Mr. Bond
  30. When Mr. Bond, the veterinarian, came to look at the black horse that lay groaning on the grass, he felt him all over, and shook his head; one of his legs was broken.
    "one of his legs was broken": (a) Mr. Bond (b) the black horse
    Answer: (b) the black horse
  31. Then he took a piece of iron the shape of the horse's foot, and clapped it on, and drove some nails through the shoe quite into its hoof, so that the shoe was firmly on
    "clapped it on": (a) piece of iron (b) shape (c) horse (d) horse's foot
    Answer: (a) piece of iron
  32. The stable was very roomy, with four good stalls; a large swinging window opened into the yard, which made it pleasant and airy.
    "made it pleasant": (a) stable (b) window (c) yard
    Answer: (a) stable
  33. We saw a boy trying to leap a pony over a gate; the pony would not take the leap, and the boy cut him with the whip, but he only turned to one side.
    "he only turned to one side": (a) boy (b) pony
    Answer: (b) pony
  34. The boy continued to whip the pony, and eventually the pony threw him over. John laughed out quite loud. "Served him right," he said.
    "Served him right": (a) boy (b) pony (c) John
    Answer: (a) boy
  35. The boy continued to whip the pony, and eventually the pony threw him over. John laughed out quite loud. "Good for him," he said.
    "Good for him": (a) boy (b) pony (c) John
    Answer: (b) pony
  36. The pony behaved well, sir, and showed no vice; but at last he just threw up his heels and tipped the young gentleman into the thorn hedge. He wanted me to help him out, but I hope you will excuse me, sir, I did not feel inclined to do so.
    "He wanted me": (a) pony (b) young gentleman
    Answer: (b) young gentleman
  37. The pony behaved well, sir, and showed no vice; but at last he just threw up his heels and tipped the young gentleman into the thorn hedge. He wanted me to help him out, but I hope you will excuse me, sir, I did not feel inclined to do so.
    "help him out": (a) pony (b) young gentleman
    Answer: (b) young gentleman
  38. After I saw Bill catching flies and pulling off their wings, I boxed his ears. I showed the master the flies, some crushed and some crawling about helpless, and I showed him the wings on the window sill. I never saw him so angry before; but as Bill was still howling and whining, like the coward that he was, he did not give him any more punishment of that kind, but set him up on a stool for the rest of the afternoon, and said that he should not go out to play for that week.
    "I showed him the wings": (a) Bill (b) the master
    Answer: (b) the master
  39. After I saw Bill catching flies and pulling off their wings, I boxed his ears. I showed the master the flies, some crushed and some crawling about helpless, and I showed him the wings on the window sill. I never saw him so angry before; but as Bill was still howling and whining, like the coward that he was, he did not give him any more punishment of that kind, but set him up on a stool for the rest of the afternoon, and said that he should not go out to play for that week.
    "I never saw him": (a) Bill (b) the master
    Answer: (b) the master
  40. After I saw Bill catching flies and pulling off their wings, I boxed his ears. I showed the master the flies, some crushed and some crawling about helpless, and I showed him the wings on the window sill. I never saw him so angry before; but as Bill was still howling and whining, like the coward that he was, he did not give him any more punishment of that kind, but set him up on a stool for the rest of the afternoon, and said that he should not go out to play for that week.
    "coward that he was": (a) Bill (b) the master
    Answer: (a) Bill
  41. After I saw Bill catching flies and pulling off their wings, I boxed his ears. I showed the master the flies, some crushed and some crawling about helpless, and I showed him the wings on the window sill. I never saw him so angry before; but as Bill was still howling and whining, like the coward that he was, he did not give him any more punishment of that kind, but set him up on a stool for the rest of the afternoon, and said that he should not go out to play for that week.
    "he did not give": (a) Bill (b) the master
    Answer: (b) the master
  42. After I saw Bill catching flies and pulling off their wings, I boxed his ears. I showed the master the flies, some crushed and some crawling about helpless, and I showed him the wings on the window sill. I never saw him so angry before; but as Bill was still howling and whining, like the coward that he was, he did not give him any more punishment of that kind, but set him up on a stool for the rest of the afternoon, and said that he should not go out to play for that week.
    "give him any more punishment": (a) Bill (b) the master
    Answer: (a) Bill
  43. After I saw Bill catching flies and pulling off their wings, I boxed his ears. I showed the master the flies, some crushed and some crawling about helpless, and I showed him the wings on the window sill. I never saw him so angry before; but as Bill was still howling and whining, like the coward that he was, he did not give him any more punishment of that kind, but set him up on a stool for the rest of the afternoon, and said that he should not go out to play for that week.
    "set him up on a stool": (a) Bill (b) the master
    Answer: (a) Bill
  44. After I saw Bill catching flies and pulling off their wings, I boxed his ears. I showed the master the flies, some crushed and some crawling about helpless, and I showed him the wings on the window sill. I never saw him so angry before; but as Bill was still howling and whining, like the coward that he was, he did not give him any more punishment of that kind, but set him up on a stool for the rest of the afternoon, and said that he should not go out to play for that week.
    "he should not go out": (a) Bill (b) the master
    Answer: (a) Bill
  45. Sir Clifford wants me to find him a new groom, about twenty or twenty-one, who knows his business. His old coachman is getting feeble, and he wants a man to work with him and get into his ways, who would be able, when the old man was pensioned off, to step into his place.
    "knows his business": (a) Sir Clifford (b) new groom
    Answer: (b) new groom
  46. Sir Clifford wants me to find him a new groom, about twenty or twenty-one, who knows his business. His old coachman is getting feeble, and he wants a man to work with him and get into his ways, who would be able, when the old man was pensioned off, to step into his place.
    "His old coachman": (a) Sir Clifford (b) new groom
    Answer: (a) Sir Clifford
  47. Sir Clifford wants me to find him a new groom, about twenty or twenty-one, who knows his business. His old coachman is getting feeble, and he wants a man to work with him and get into his ways, who would be able, when the old man was pensioned off, to step into his place.
    "he wants a man": (a) Sir Clifford (b) new groom (c) old coachman
    Answer: (a) Sir Clifford
    Note: Arguably, (c) old coachman could also work as an answer. Ambiguous questions of this sort will not occur in the competition.
  48. Sir Clifford wants me to find him a new groom, about twenty or twenty-one, who knows his business. His old coachman is getting feeble, and he wants a man to work with him and get into his ways, who would be able, when the old man was pensioned off, to step into his place.
    "work with him": (a) Sir Clifford (b) new groom (c) old coachman
    Answer: (c) old coachman
  49. Sir Clifford wants me to find him a new groom, about twenty or twenty-one, who knows his business. His old coachman is getting feeble, and he wants a man to work with him and get into his ways, who would be able, when the old man was pensioned off, to step into his place.
    "get into his ways": (a) Sir Clifford (b) new groom (c) old coachman
    Answer: (c) old coachman
  50. Sir Clifford wants me to find him a new groom, about twenty or twenty-one, who knows his business. His old coachman is getting feeble, and he wants a man to work with him and get into his ways, who would be able, when the old man was pensioned off, to step into his place.
    "step into his place": (a) Sir Clifford (b) new groom (c) old coachman
    Answer: (c) old coachman
  51. My master and mistress decided to pay a visit to some friends who lived about forty-six miles from our home, and James was to drive them.
    "drive them": (a) master and mistress (b) some friends
    Answer: (a) master and mistress
  52. A man's life and a horse's life are worth more than some foxes' tails; al least they ought to be.
    "they ought to be": (a) a man's life and a horse's life (b) foxes' tails
    Answer: (a) a man's life and a horse's life
  53. When my parents died, I was not earning enough to keep myself or my crippled sister Nelly, and without our kind mistress, we would have gone to the poorhouse. Nelly calls her her angel and she has good right to do so.
    "Nelly calls her": (a) Nelly (b) mistress
    Answer: (b) mistress
  54. When my parents died, I was not earning enough to keep myself or my crippled sister Nelly, and without our kind mistress, we would have gone to the poorhouse. Nelly calls her her angel and she has good right to do so.
    "her angel ": (a) Nelly (b) mistress
    Answer: (a) Nelly
  55. When my parents died, I was not earning enough to keep myself or my crippled sister Nelly, and without our kind mistress, we would have gone to the poorhouse. Nelly calls her her angel and she has good right to do so.
    "she has good right":
    Answer: (a) Nelly
  56. Our mistress hired a room for my crippled sister Nelly, and she gave her knitting and needlework when she was able to do it, and when she was ill she sent her dinners and many nice comfortable things and was like a mother to her
    "she gave": (a) mistress (b) Nelly
    Answer: (a) mistress
  57. Our mistress hired a room for my crippled sister Nelly, and she gave her knitting and needlework when she was able to do it, and when she was ill she sent her dinners and many nice comfortable things and was like a mother to her
    "gave her knitting": (a) mistress (b) Nelly
    Answer: (b) Nelly
  58. Our mistress hired a room for my crippled sister Nelly, and she gave her knitting and needlework when she was able to do it, and when she was ill she sent her dinners and many nice comfortable things and was like a mother to her
    "when she was able": (a) mistress (b) Nelly
    Answer: (b) Nelly
  59. Our mistress hired a room for my crippled sister Nelly, and she gave her knitting and needlework when she was able to do it, and when she was ill she sent her dinners and many nice comfortable things and was like a mother to her
    "when she was ill": (a) mistress (b) Nelly
    Answer: (b) Nelly
  60. Our mistress hired a room for my crippled sister Nelly, and she gave her knitting and needlework when she was able to do it, and when she was ill she sent her dinners and many nice comfortable things and was like a mother to her
    "she sent": (a) mistress (b) Nelly
    Answer: (a) mistress
  61. Our mistress hired a room for my crippled sister Nelly, and she gave her knitting and needlework when she was able to do it, and when she was ill she sent her dinners and many nice comfortable things and was like a mother to her
    "sent her dinners": (a) mistress (b) Nelly
    Answer: (b) Nelly
  62. Our mistress hired a room for my crippled sister Nelly, and she gave her knitting and needlework when she was able to do it, and when she was ill she sent her dinners and many nice comfortable things and was like a mother to her
    "like a mother to her": (a) mistress (b) Nelly
    Answer: (b) Nelly

WSC-I Challenge Disambiguation Problems

The full set of problems in the first Winograd Schema Challenge, held at IJCAI-16, can be found here

For questions or comments about commonsensereasoning.org please email leora@cs.nyu.edu.

Website design by Benjamin Johnston, based on the Fluid 960 Grid System by Stephen Bau and Nathan Smith. Crowd photo by James Cridland.