Following are two lists: one of sample Pronoun Disambiguation Problems (PDPs), and a list of the challenge PDPs used in the First Winograd Schema Challenge held at IJCAI 2016 .
The passages are taken or modified from existing published works. The sample problems are taken from Carolyn Keene: Nancy Drew: The Quest of the Missing Map, Grosset & Dunlap, 1969, 1942; Jean de Brunhoff: The Story of Babar, Random House, 1933; and Anna Sewell: Black Beauty, J. M. Lupton, 1877. Sentences in WSC-2016 were taken from Laura Ingalls Wilder Little Town on the Prarie, P.G. Woodhouse: Carry on, Jeeves , Sidney Taylor: All of a Kind Family, Esther Forbes: Johnny Tremaine, Ruth Sawyer: The Year of Jubilo, and several other sources.
Sentences have been modified to summarize context and backstory, which are often given over several paragraphs, into a single sentence (PDP 33); to clarify or simplify (PDP 22); or to change gender or number of nouns and pronouns in order to introduce ambiguity (PDP 2). For challenge problems, character names were also changed. Note that a single passage may give rise to multiple Pronoun Disambiguation Problems.
Sample Pronoun Disambiguation Problems
The pronoun to be disambiguated is highlighted in bold. It is repeated again, with a snippet of context, and with several candidate answers, in the line following the passage. The correct answer is given in the next line.
-
While Nancy and Ellen counted the silverware, Mrs. Smith hastened upstairs. In a few minutes she returned and one look at her stricken face told the girls that the precious map was gone.
"her stricken face": a) Nancy (b) Ellen (c) Mrs. Smith
Answer: (c) Mrs. Smith
-
Sergeant Holmes asked the girls to describe the intruder. Nancy not only provided the policeman with an excellent description of the heavyset thirty-year-old prowler, but drew a rough sketch of his face.
"rough sketch of his face": (a) Sergeant Holmes (b) the intruder
Answer: (b) the intruder
-
After a somewhat lengthy conversation, Nancy convinced the housekeeper that there was no immediate cause for alarm. She did not mention the incident at the Hamilton bus station, knowing it would only add to Mrs. Gruen’s uneasiness.
"She did not mention": (a) Nancy (b) the housekeeper
Answer: (a) Nancy
-
I asked Dave to get me my sweater from the other side of the yacht. While he was gone, I rested my arm on the rail over there and suddenly it gave way.
"it gave way": (a) sweater (b) yacht (c) arm (d) rail
Answer: (d) rail
-
Do you suppose that Peter is responsible for the captain’s illness? Maybe he bribed the cook to put something in his food.
"he bribed the cook": (a) Peter (b) the captain
Answer: (a) Peter
-
Do you suppose that Peter is responsible for the captain’s illness? Maybe he bribed the cook to put something in his food.
"his food": (a) Peter (b) the captain (c) the cook
Answer: (b) the captain
-
Babar wonders how he can get new clothing. Luckily, a very rich old man who has always been fond of little elephants understands right away that he is longing for a fine suit. As he likes to make people happy, he gives him his wallet.
"he is longing for a fine suit": (a) Babar (b) old man
Answer: (a) Babar
-
Babar wonders how he can get new clothing. Luckily, a very rich old man who has always been fond of little elephants understands right away that he is longing for a fine suit. As he likes to make people happy, he gives him his wallet.
"he likes to make people happy": (a) Babar (b) old man
Answer: (b) old man
-
Babar wonders how he can get new clothing. Luckily, a very rich old man who has always been fond of little elephants understands right away that he is longing for a fine suit. As he likes to make people happy, he gives him his wallet.
"he gives": (a) Babar (b) old man
-
Babar wonders how he can get new clothing. Luckily, a very rich old man who has always been fond of little elephants understands right away that he is longing for a fine suit. As he likes to make people happy, he gives him his wallet.
"gives him": (a) Babar (b) old man
Answer: (a) Babar
-
Babar wonders how he can get new clothing. Luckily, a very rich old man who has always been fond of little elephants understands right away that he is longing for a fine suit. As he likes to make people happy, he gives him his wallet.
"his wallet": (a) Babar (b) old man
Answer: (b) old man
-
Well satisfied with his purchases and feeling very elegant indeed, Babar goes to the photographer to have his picture taken.
"his picture": (a) Babar (b) the photographer
Answer: (a) Babar
-
Meanwhile, in the forest, the elephants are calling and hunting high and low for Arthur and Celeste, and their mothers are very worried. Fortunately, in flying over the town, an old marabou bird has seen them and come back quickly to tell the news.
"their mothers": (a) the elephants (b) Arthur and Celeste
Answer: (b) Arthur and Celeste
-
Meanwhile, in the forest, the elephants are calling and hunting high and low for Arthur and Celeste, and their mothers are very worried. Fortunately, in flying over the town, an old marabou bird has seen them and come back quickly to tell the news.
"has seen them": (a) the elephants (b) Arthur and Celeste (c) their mothers
Answer: (b) Arthur and Celeste
-
The mothers of Arthur and Celeste have come to the town to fetch them. They are very happy to have them back, but they scold them just the same because they ran away.
"fetch them": (a) mothers (b) Arthur and Celeste
Answer: (b) Arthur and Celeste
-
The mothers of Arthur and Celeste have come to the town to fetch them. They are very happy to have them back, but they scold them just the same because they ran away.
"They are very happy": (a) mothers (b) Arthur and Celeste
Answer: (a) mothers
-
The mothers of Arthur and Celeste have come to the town to fetch them. They are very happy to have them back, but they scold them just the same because they ran away.
"have them back": (a) mothers (b) Arthur and Celeste
Answer: (b) Arthur and Celeste
-
The mothers of Arthur and Celeste have come to the town to fetch them. They are very happy to have them back, but they scold them just the same because they ran away.
"they scold": (a) mothers (b) Arthur and Celeste
Answer: (a) mothers
-
The mothers of Arthur and Celeste have come to the town to fetch them. They are very happy to have them back, but they scold them just the same because they ran away.
"scold them": (a) mothers (b) Arthur and Celeste
Answer: (b) Arthur and Celeste
-
The mothers of Arthur and Celeste have come to the town to fetch them. They are very happy to have them back, but they scold them just the same because they ran away.
"they ran away": (a) mothers (b) Arthur and Celeste
Answer: (b) Arthur and Celeste
-
The first place that I can well remember was a large pleasant meadow with a pond of clear water in it. Some shady trees leaned over it, and rushes and water-lilies grew at the deep end.
"leaned over it": (a) meadow (b) pond
Answer: (b) pond
-
One day Dick was teasing the colts, and did not know that the master was in the next field; but he was there, watching what was going on; over the hedge he jumped in a snap, and catching Dick by the arm, he hit him so hard that he roared with the pain and surprise.
"he was there": (a) Dick (b) the master
Answer: (b) the master
-
One day Dick was teasing the colts, and did not know that the master was in the next field; but he was there, watching what was going on; over the hedge he jumped in a snap, and catching Dick by the arm, he hit him so hard that he roared with the pain and surprise.
"he jumped ": (a) Dick (b) the master
Answer: (b) the master
-
One day Dick was teasing the colts, and did not know that the master was in the next field; but he was there, watching what was going on; over the hedge he jumped in a snap, and catching Dick by the arm, he hit him so hard that he roared with the pain and surprise.
"he gave": (a) Dick (b) the master
Answer: (b) the master
-
One day Dick was teasing the colts, and did not know that the master was in the next field; but he was there, watching what was going on; over the hedge he jumped in a snap, and catching Dick by the arm, he hit him so hard that he roared with the pain and surprise.
"hit him so hard": (a) Dick (b) the master
Answer: (a) Dick
-
One day Dick was teasing the colts, and did not know that the master was in the next field; but he was there, watching what was going on; over the hedge he jumped in a snap, and catching Dick by the arm, he hit him so hard that
he roared with the pain and surprise.
"that he roared": (a) Dick (b) the master
Answer: (a) Dick
-
When Mr. Bond, the veterinarian, came to look at the black horse that lay groaning on the grass, he felt him all over, and shook his head; one of his legs was broken.
"he felt": (a) Mr. Bond (b) the black horse
Answer: (a) Mr. Bond
-
When Mr. Bond, the veterinarian, came to look at the black horse that lay groaning on the grass, he felt him all over, and shook his head; one of his legs was broken.
"felt him all over": (a) Mr. Bond (b) the black horse
Answer: (b) the black horse
-
When Mr. Bond, the veterinarian, came to look at the black horse that lay groaning on the grass, he felt him all over, and shook his head; one of his legs was broken.
"shook his head": (a) Mr. Bond (b) the black horse
Answer: (a) Mr. Bond
-
When Mr. Bond, the veterinarian, came to look at the black horse that lay groaning on the grass, he felt him all over, and shook his head; one of his legs was broken.
"one of his legs was broken": (a) Mr. Bond (b) the black horse
Answer: (b) the black horse
-
Then he took a piece of iron the shape of the horse's foot, and clapped it on, and drove some nails through the shoe quite into its hoof, so that the shoe was firmly on
"clapped it on": (a) piece of iron (b) shape (c) horse (d) horse's foot
Answer: (a) piece of iron
-
The stable was very roomy, with four good stalls; a large swinging window opened into the yard, which made it pleasant and airy.
"made it pleasant": (a) stable (b) window (c) yard
Answer: (a) stable
-
We saw a boy trying to leap a pony over a gate; the pony would not take the leap, and the boy cut him with the whip, but he only turned to one side.
"he only turned to one side": (a) boy (b) pony
Answer: (b) pony
-
The boy continued to whip the pony, and eventually the pony threw him over. John laughed out quite loud. "Served him right," he said.
"Served him right": (a) boy (b) pony (c) John
Answer: (a) boy
-
The boy continued to whip the pony, and eventually the pony threw him over. John laughed out quite loud. "Good for him," he said.
"Good for him": (a) boy (b) pony (c) John
Answer: (b) pony
-
The pony behaved well, sir, and showed no vice; but at last he just threw up his heels and tipped the young gentleman into the thorn hedge. He wanted me to help him out, but I hope you will excuse me, sir, I did not feel inclined to do so.
"He wanted me": (a) pony (b) young gentleman
Answer: (b) young gentleman
-
The pony behaved well, sir, and showed no vice; but at last he just threw up his heels and tipped the young gentleman into the thorn hedge. He wanted me to help him out, but I hope you will excuse me, sir, I did not feel inclined to do so.
"help him out": (a) pony (b) young gentleman
Answer: (b) young gentleman
-
After I saw Bill catching flies and pulling off their wings, I boxed his ears. I showed the master the flies, some crushed and some crawling about helpless, and I showed him the wings on the window sill. I never saw him so angry before; but as Bill was still howling and whining, like the coward that he was, he did not give him any more punishment of that kind, but set him up on a stool for the rest of the afternoon, and said that he should not go out to play for that week.
"I showed him the wings": (a) Bill (b) the master
Answer: (b) the master
-
After I saw Bill catching flies and pulling off their wings, I boxed his ears. I showed the master the flies, some crushed and some crawling about helpless, and I showed him the wings on the window sill. I never saw him so angry before; but as Bill was still howling and whining, like the coward that he was, he did not give him any more punishment of that kind, but set him up on a stool for the rest of the afternoon, and said that he should not go out to play for that week.
"I never saw him": (a) Bill (b) the master
Answer: (b) the master
-
After I saw Bill catching flies and pulling off their wings, I boxed his ears. I showed the master the flies, some crushed and some crawling about helpless, and I showed him the wings on the window sill. I never saw him so angry before; but as Bill was still howling and whining, like the coward that he was, he did not give him any more punishment of that kind, but set him up on a stool for the rest of the afternoon, and said that he should not go out to play for that week.
"coward that he was": (a) Bill (b) the master
Answer: (a) Bill
-
After I saw Bill catching flies and pulling off their wings, I boxed his ears. I showed the master the flies, some crushed and some crawling about helpless, and I showed him the wings on the window sill. I never saw him so angry before; but as Bill was still howling and whining, like the coward that he was, he did not give him any more punishment of that kind, but set him up on a stool for the rest of the afternoon, and said that he should not go out to play for that week.
"he did not give": (a) Bill (b) the master
Answer: (b) the master
-
After I saw Bill catching flies and pulling off their wings, I boxed his ears. I showed the master the flies, some crushed and some crawling about helpless, and I showed him the wings on the window sill. I never saw him so angry before; but as Bill was still howling and whining, like the coward that he was, he did not give him any more punishment of that kind, but set him up on a stool for the rest of the afternoon, and said that he should not go out to play for that week.
"give him any more punishment": (a) Bill (b) the master
Answer: (a) Bill
-
After I saw Bill catching flies and pulling off their wings, I boxed his ears. I showed the master the flies, some crushed and some crawling about helpless, and I showed him the wings on the window sill. I never saw him so angry before; but as Bill was still howling and whining, like the coward that he was, he did not give him any more punishment of that kind, but set him up on a stool for the rest of the afternoon, and said that he should not go out to play for that week.
"set him up on a stool": (a) Bill (b) the master
Answer: (a) Bill
-
After I saw Bill catching flies and pulling off their wings, I boxed his ears. I showed the master the flies, some crushed and some crawling about helpless, and I showed him the wings on the window sill. I never saw him so angry before; but as Bill was still howling and whining, like the coward that he was, he did not give him any more punishment of that kind, but set him up on a stool for the rest of the afternoon, and said that he should not go out to play for that week.
"he should not go out": (a) Bill (b) the master
Answer: (a) Bill
-
Sir Clifford wants me to find him a new groom, about twenty or twenty-one, who knows his business. His old coachman is getting feeble, and he wants a man to work with him and get into his ways, who would be able, when the old man was pensioned off, to step into his place.
"knows his business": (a) Sir Clifford (b) new groom
Answer: (b) new groom
-
Sir Clifford wants me to find him a new groom, about twenty or twenty-one, who knows his business. His old coachman is getting feeble, and he wants a man to work with him and get into his ways, who would be able, when the old man was pensioned off, to step into his place.
"His old coachman": (a) Sir Clifford (b) new groom
Answer: (a) Sir Clifford
-
Sir Clifford wants me to find him a new groom, about twenty or twenty-one, who knows his business. His old coachman is getting feeble, and he wants a man to work with him and get into his ways, who would be able, when the old man was pensioned off, to step into his place.
"he wants a man": (a) Sir Clifford (b) new groom (c) old coachman
Answer: (a) Sir Clifford
Note: Arguably, (c) old coachman could also work as an answer. Ambiguous questions of this sort will not occur in the competition.
-
Sir Clifford wants me to find him a new groom, about twenty or twenty-one, who knows his business. His old coachman is getting feeble, and he wants a man to work with him and get into his ways, who would be able, when the old man was pensioned off, to step into his place.
"work with him": (a) Sir Clifford (b) new groom (c) old coachman
Answer: (c) old coachman
-
Sir Clifford wants me to find him a new groom, about twenty or twenty-one, who knows his business. His old coachman is getting feeble, and he wants a man to work with him and get into his ways, who would be able, when the old man was pensioned off, to step into his place.
"get into his ways": (a) Sir Clifford (b) new groom (c) old coachman
Answer: (c) old coachman
-
Sir Clifford wants me to find him a new groom, about twenty or twenty-one, who knows his business. His old coachman is getting feeble, and he wants a man to work with him and get into his ways, who would be able, when the old man was pensioned off, to step into his place.
"step into his place": (a) Sir Clifford (b) new groom (c) old coachman
Answer: (c) old coachman
-
My master and mistress decided to pay a visit to some friends who lived about forty-six miles from our home, and James was to drive them.
"drive them": (a) master and mistress (b) some friends
Answer: (a) master and mistress
-
A man's life and a horse's life are worth more than some foxes' tails; al least they ought to be.
"they ought to be": (a) a man's life and a horse's life (b) foxes' tails
Answer: (a) a man's life and a horse's life
-
When my parents died, I was not earning enough to keep myself or my crippled sister Nelly, and without our kind mistress, we would have gone to the poorhouse. Nelly calls her her angel and she has good right to do so.
"Nelly calls her": (a) Nelly (b) mistress
Answer: (b) mistress
-
When my parents died, I was not earning enough to keep myself or my crippled sister Nelly, and without our kind mistress, we would have gone to the poorhouse. Nelly calls her her angel and she has good right to do so.
"her angel ": (a) Nelly (b) mistress
Answer: (a) Nelly
-
When my parents died, I was not earning enough to keep myself or my crippled sister Nelly, and without our kind mistress, we would have gone to the poorhouse. Nelly calls her her angel and she has good right to do so.
"she has good right":
Answer: (a) Nelly
-
Our mistress hired a room for my crippled sister Nelly, and she gave her knitting and needlework when she was able to do it, and when she was ill she sent her dinners and many nice comfortable things and was like a mother to her
"she gave": (a) mistress (b) Nelly
Answer: (a) mistress
-
Our mistress hired a room for my crippled sister Nelly, and she gave her knitting and needlework when she was able to do it, and when she was ill she sent her dinners and many nice comfortable things and was like a mother to her
"gave her knitting": (a) mistress (b) Nelly
Answer: (b) Nelly
-
Our mistress hired a room for my crippled sister Nelly, and she gave her knitting and needlework when she was able to do it, and when she was ill she sent her dinners and many nice comfortable things and was like a mother to her
"when she was able": (a) mistress (b) Nelly
Answer: (b) Nelly
-
Our mistress hired a room for my crippled sister Nelly, and she gave her knitting and needlework when she was able to do it, and when she was ill she sent her dinners and many nice comfortable things and was like a mother to her
"when she was ill": (a) mistress (b) Nelly
Answer: (b) Nelly
-
Our mistress hired a room for my crippled sister Nelly, and she gave her knitting and needlework when she was able to do it, and when she was ill she sent her dinners and many nice comfortable things and was like a mother to her
"she sent": (a) mistress (b) Nelly
Answer: (a) mistress
-
Our mistress hired a room for my crippled sister Nelly, and she gave her knitting and needlework when she was able to do it, and when she was ill she sent her dinners and many nice comfortable things and was like a mother to her
"sent her dinners": (a) mistress (b) Nelly
Answer: (b) Nelly
-
Our mistress hired a room for my crippled sister Nelly, and she gave her knitting and needlework when she was able to do it, and when she was ill she sent her dinners and many nice comfortable things and was like a mother to her
"like a mother to her": (a) mistress (b) Nelly
Answer: (b) Nelly
WSC-I Challenge Disambiguation Problems
The full set of problems in the first Winograd Schema Challenge, held at IJCAI-16, can be found here