When I met yesterday with IBM's chief scientist behind Jeopardy, David Ferrucci, he was wearing a Toronto Blue Jays jacket. It had to do with Watson's only signficant blooper in an otherwise dominant performance in the second half of its first game.
The Final Jeopardy category was U.S. Cities. The clue: "Its largest airport is named for a World War II hero, its second largest for a World War II battle." Watson, strangely, came up with the response: "What is Toronto??????" It was programmed to add all those question marks to show the audience that it had very low confidence in the response. But still, how could it choose Toronto in a category for U.S. cities.
After the game, Ferrucci and his team were eager to explain Watson's thinking process. Strangely, from a PR point of view, they seemed determined to focus on one moment of weakness in session that exhibited Watson's strengths. But they have poured four years of research into this machine, and they like to clear up doubts.
A few key issues:
1) Watson can never be sure of anything. Is it possible that the old rock star Alice Cooper is a man? If Watson finds enough evidence, it will bet on it--even though the name "Alice" is sure to create a lot of doubt. This flexibility in its thinking can save Watson from gaffes--but also lead to a few.
2) Category titles cannot be trusted. I blogged about this earlier, in a post How Watson Thinks. It has learned through exhaustive statistical analysis that many clues do not jibe with categories. A category about US novelists, for example, can ask about J.D. Salinger's masterpiece. Catcher in the Rye is a novel, not a novelist! These things happen time and again, and Watson notices. So it pays scant attention to the categories.
3) If this had been a normal Jeopardy clue, Watson would not have buzzed. It had only 14% confidence in Toronto (whose Pearson airport is named for a World War II hero), and 11% in Chicago. Watson simply did not come up with the answer, and Toronto was its guess.
Even so, how could it guess that Toronto was an American city? Here we come to the weakness of statistical analysis. While searching through data, it notices that the United States is often called America. Toronto is a North American city. Its baseball team, the Blue Jays, plays in the American League. (That's why Ferrucci was wearing a Blue Jay jacket). If Watson happened to study the itinerary of my The Numerati book tour, it included a host of American cities, from Philadelphia and Pittsburgh, to Seattle, San Francisco, and Toronto. In documents like that, people often don't stop to note for inquiring computers that Toronto actually shouldn't be placed in the group.
Long story short: Watson blew it. It happens. It earned it lots of abuse on Twitter, such as:
lutheranish
Watson the computer is a dumbass. Doesn't even know that Toronto is not in the US. #jeopardy...
But now, hopefully, Lutheranish and other critics will understand why.
