Tuesday, June 21

Inside ACL: Building Watson DeepQA keynote Address by David Ferrucci

This morning David Ferrucci gave the Association for Computation Linguistics (ACL) 2011 keynote talk. Michael Bendersky is attending the conference and was very generous to send me his notes on the first keynote talk. Be sure to read his paper, Joint Annotation of Search Queries. Here are his notes from the talk,

Building Watson: An Overview of the DeepQA Project
  • What’s the difference between playing chess and understanding human language?
    - People find chess difficult and natural language easy
    - Many non-scientists don’t realize how difficult human language understanding really is

  • Computers are good at
    - Understanding formulas
    - Understanding structured query languages

  • Computers are bad at
    - Parsing ambiguous natural language

  • The system challenges
    - Open domain
    - Complex language
    - High precision
    - Accurate confidence – only buzz in when you’re very confident
    - High speed

  • Core technologies
    - Deep parsing – using a proprietary IBM technology that has been developed over the last 20 years
    - Relation detection
    - Multiple parse interpretations
    - Multiple query formulations per parse

  • Co-reference resolution
    - The entire research was driven by a single end-to-end metric – how much the proposed solution improves the Jeopardy game
    - Some improvements on a single algorithm might be redundant or harmful in the overall solution

  • Jeopardy is open-domain – not using ontologies that were crafted specifically for Jeopardy
    - Using general resources: Wordnet, YAGO

  • Learning from Reading
    - Parsing sentences in the text
    - Generalization and Statistical Aggregation

  • Some questions require decomposition and synthesis
    - Using techniques to decompose questions into parts
    - Synthesis of answers from different parts
    - Helps in answering questions that involve puns/rhyming

  • Some questions require finding a missing link between concepts
    - Using spreading activation to find links
    - eg, link between “shirt”, “tv remote”, “telephone” -> buttons

  • Metrics for performance evaluation
    - Plot x- % answered, y – Precision
    - Winners clouds – answered at least 50% of the questions, precision 80-92%
    - The goal was to get Watson into the winner cloud – achieved and went over the cloud by the Jeopardy game

  • Great leaps in performance from 2007. In the beginning, breaking even in the game seemed like an accomplishment

  • Watson is self-contained. Deciding what content to use is very hard – the amount of hardware is limited.

  • Guidelines
    - Specific large hand-crafted methods won’t cut it
    - Combining intelligence from diverse methods using machine learning techniques
    - Massive Parallelism is a Key Enabler

  • DeepQA – QA system underlying Watson
    - Many components for parsing and multiple answer generation
    - Logistic regression to weight the different features and rank the answers

  • Search systems used: Indri & Lucene. Both were modified to reduce run-time

  • Work process
    - All group members working in the same open space room
    - NLP researchers, IR researchers, ML researchers, linguists, statisticians
    - 8,000 experiments – all documented with tools that allow analysis by question/algorithm/features

  • Run-time
    - Single CPU time for answering a question – 2 hours
    - Scaled out to 3,000 CPU’s – 2-3 seconds
    - Enabled by the built-in parallelization of the algorithms
What I find particularly striking is the deep analysis of a contained corpus, particularly the analysis to find various kinds of missing links. The hardware is limited and the corpus is very circumscribed in order to run complex and expensive algorithms - and it results in significant improvements!
  • How would you develop a system for the real-time web where what's meaningful is constantly in flux?
Ultimately, the true test of DeepQA will be how it generalizes to domains beyond Jeopardy. I hope this is just the beginning for Watson.
Thanks again to Michael for his notes. Look for more highlights from ACL coming soon!