Jeopardy! Day 3 Update
If you are a technology person, you had to be impressed. We all know who won by now so I won’t belabor it. Ken Jennings played better and made a game of it … at least for a while. He seemed to anticipate the buzz a little bit better and got on a roll.
You may have noticed that Watson struggled in certain categories last night. “Actors Who Direct” gave very short clues (or questions) like “The Great Debaters” for which the correct answer was “Who is Denzel Washington”. For Watson, the longer the question, the better. If it takes a longer time for Alex to read the question, Watson has more time to consider candidate answers, evidence scores and confidence rankings. This is another reason why Watson does better in certain categories.
In an attempt to remain competitive in this situation, Watson has multiple ways to process clues or questions. There is what is called the “short path” (to an answer). This is used for shorter questions when Watson has less time to decide whether to buzz in or not. Watson is more inconsistent when it has to answer faster. As seen last night, he either chose not to answer or Ken and Brad beat him to it.
In the end, the margin of victory was decisive for Watson. In total, $1.25 million was donated to charity and Ken and Brad took home parting gifts of $150,000 and $100,000 respectively … pretty good for all involved. The real winners are science and technology. This is a major advance in computing that could revolutionize the way we interact with computers … especially with questions and answers. The commercial applications seem endless.
Jeopardy! Day 2 Update
Last night was compelling to watch. I was at the Washington, DC viewing event with several hundred customers, partners and IBMers. The atmosphere in the briefing center was electric. When the game started with Watson taking command, the room erupted in cheers. After Watson got on a roll and steamrolled Brad and Ken for most of Double Jeopardy, the room began to grow silent in awe of what was happening.
Erik Mueller (IBM Research) was our featured speaker. He was bombarded … before, during and after the match with questions like “How does he know what to bet?” “How does Watson process text?” How would this be used in medical research?” “What books were in Watson’s knowledge base?” “Can Watson hear?” “Does he have to press a button like the human contestants?” and many more.
I was there as a subject matter expert and even though the spotlight was rightfully on Eric, I did get to answer a question on how some of Watson’s technology was being used today. I explained how our IBM Content Analytics is used and how it is helping to power Watson’s natural language prowess.
When Watson incorrectly answered “What is Toronto?” in Final Jeopardy, the room audibly gasped (myself included). As everyone seemed to hold their breath, I looked at Erik and he was smiling like a Cheshire cat … brimming with confidence. The room cheered and applauded when Watson’s small bet was revealed … a seeming acknowledgment to the technological brilliance. Applause for a wrong answer!
Afterward, there were many ideas on how Watson could be applied. My favorite was from a legal industry colleague who had a number of suggestions for how Watson could optimize document review and analysis that is currently a problem for judges and litigators.
Yesterday I said humans have a slight advantage. And while Watson has built an impressive lead, I still feel that way. Many of yesterday’s categories played to Watson’s fact-based strengths. It could go the other way tonight and Brad and Ken could get right back into the match. The second game will air tonight in its entirety and the scores from both games will be combined to determine the $1 million prize winner.
Watson is entering tonight with a more than $25,000 lead. IBM is donating all prize winnings to charity and Ken Jennings and Brad Rutter are donating 50% to charity.
Jeopardy! Day 1 Update
After Day 1, Watson is tied with Brad Rutter at $5,000 going into Double Jeopardy – which is pretty impressive. Ken Jennings has yet to catch his stride. Brad and Ken seemed a little shell-shocked at first, but Brad rebounded right when Watson was faltering towards the end of the first round. This got me thinking... Should I go into a little more detail about who really has the advantage … Watson or the humans?
If you watched it last night, you may have observed that Watson does very well with factual questions. He did very well in the Beatles song category – they were mostly facts with contextual references to lyrics. Answers that involve multiple facts, all of which are required to answer the correct response but are unlikely to be found in the same place, are much harder for Watson. This is why Watson missed the Harry Potter question involving Lord Voldemort. Watson also switched categories frequently which is part of his game strategy. You may have also noticed that Watson can’t see or hear. He answered a question wrong even though Ken gave the same wrong answer seconds before. More on this later in the post.
Here goes … my take on who has the advantage …
Question Understanding: Advantage Humans
Humans: Seemingly Effortless. Almost instantly knows what is being asked, what is important and how it applies – very naturally gets focus, references, hints, puns, implications, etc.
Watson: Hugely Challenging. Has to be programmed to analyze enormous numbers of possibilities to get just a hint of the relevant meaning. Very difficult due to variability, implicit context, the ambiguity of structure and meaning in language.
Language Understanding: Advantage Humans
Humans: Seemingly Effortless. Powerful, general, deep and fast in understanding language – reading, experiencing, summarizing, storing knowledge in natural language. This information is written for human consumption so reading and understanding what it says is natural for humans.
Watson: Hugely Challenging. Answers need to be determined and justified in natural language sources like news articles, reference texts, plays, novels, etc. Watson must be carefully programmed and automatically trained to deeply analyze even just tiny subsets of language effectively. Very different from web search, must find a precise answer and understand enough of what it read to know if and why a possible answer may be correct.
Self‐Knowledge (Confidence): Advantage Humans
Humans: Seemingly Effortless. Most often, and almost instantly, humans know if they know the answer.
Watson: Hugely Challenging. 1000’s of algorithms run in parallel to find and analyze 1000’s of written texts for many different types of evidence. The results are combined, scored and weighed for their relative importance – how much they justify a candidate's answer. This has to happen in 3 seconds to compute confidence and decide whether or not to ring in before it is too late.
Breadth of Knowledge: Advantage Humans
Humans: Limited by self-contained memory. Estimates of >1000’s of terabytes are all much higher than Watson’s memory capacity. The ability to flexibly understand and summarize human relevance means that humans’ raw input capacity is even higher.
Watson: Limited by self‐contained memory. Roughly 1 Million books worth of content stored and processed in 15 Terabytes of working memory. Weaker ability to meaningfully understand, relate and summarize human‐relevant content. Must look at lots of data to compute statistical relevance.
Processing Speed: Advantage Humans
Humans: Fast Accurate Language Processing. Native, strong, fast, language abilities. Highly associative, highly flexible memory and speedy recall. Very fast to speed read clue, accurately grasp question, determine confidence and answer – in just seconds.
Watson: Hugely Challenging. On 1 CPU Watson can take over 2 hours to answer to a typical Jeopardy! question. Watson must be parallelized, perhaps in ways similar to the brain, to simultaneously use 1000’s of computing cores to compete against humans in the 3-5 second range.
Reaction Speed: Toss-up
Humans: Times the Buzz. Slower raw reaction speed but potentially faster to the buzz. Listens to clue and anticipates when to buzz in. “Timing the buzz” like this provides humans with the fastest absolute possible response time.
Watson: Fast Hand. More consistently deliver’s a fast reaction time but ONLY IF and WHEN can determine high enough confidence in time to buzz‐in. Not able to anticipate when to buzz in based on listening to clues, which gives the fastest possible response time to humans. Also has to press the same mechanical button as humans do.
Compute Power: Won’t Impact Outcome
Humans: Requires 1 brain that fits in a shoebox, can run on a tuna‐fish sandwich and be cooled with a hand‐held paper fan.
Watson: Hugely Challenging. Needs 2,880 compute cores (10 refrigerators worth in size and space) requiring about 80Kw of power and 20 tons of cooling.
Betting and Strategy: Advantage Watson
Humans: Slower, typically less precise. Uses strategy and adjusts based on situation and game position.
Watson: Faster, more accurate calculations. Uses strategy and adjusts based on situation and game position.
Emotions: Advantage Watson
Humans: Yes. Can slow down and /or confuse processing.
Watson: No. Does NOT get nervous, tired, upset or psyched out (but the Watson programming team does!).
In-Game Learning: Advantage Humans
Humans: Learn very quickly from context, voice expression and (most importantly) right and wrong answers.
Watson: Watson does not have the ability to hear (speech to text). It is my understanding that Watson is “fed” the correct answer (in text) after each question so he can learn about the category even if he gets it wrong or does not answer. However, I don’t believe he is “fed” the wrong answers though. This is a disadvantage for Watson. As seen last night, it is not uncommon for him to answer with the same wrong answer as another contestant. This also happened in the sparring rounds leading up to the taping of last night's show.
As you can see things are closely matched but a slight advantage has to go to Ken and Brad.
And what about Watson’s face?
Another observation I made was how cool Watson’s avatar was. It actually expresses what he is thinking (or processing). The Watson avatar shares the graphic structure and tonality of the IBM Smarter Planet marketing campaign; a global map projection with a halo of “thought rays.” The avatar features dozens of differentiated animation states that mirror the many stages of Jeopardy! gameplay – from choosing categories and answering clues, to winning and losing, to making Daily Double wagers and playing Final Jeopardy!.
Even Watson’s level of confidence – the numeric threshold that determines whether or not Watson will buzz in to answer – is made visible. Watson’s stage presence is designed to depict the interior processes of the advanced computing system that powers it. A significant portion of the avatar consists of colored threads orbiting around a central core. The threads and thought rays that makeup Watson’s avatar change color and speed depending on what happens during the game. For example, when Watson feels confident in an answer the rays on the avatar turn green; they turn orange when Watson gets the answer wrong. You will see the avatar speed up and activate when Watson’s algorithms are working hard to answer a clue.
I’ll be glued to the TV tonight and tomorrow. Regardless of the outcome, this whole experience has been fascinating to me.
You can also visit my previous blog postings on Watson at:
#Sedona #NLP #email #textanalytics #archiving #WikiLeaks #security #compliance #SNIA #ECM #CGOC #IBMInfoGov #Centennial #AIIM #IMRM #enterprisesearch #RIM #GAO #ARMA #ILM #IBMWatson #privacy #eDiscovery #cloud #IRM #analytics #classification #categorization #Watson #ACM #productmarketing #infogov #QA #Jeopardy #IBMECM #cloudcomputing #search