Monday, July 20

Susan Dumais: SIGIR 2009 Salton Award Keynote Talk

See also Jon's notes.

- I'm live at the SIGIR 2009 conference. Liz Liddy is introducing Susan Dumais as the Salton Award winner.

- Sue is now on stage giving her presentation:

  • Sue’s research is interdisciplinary, at the intersection of IR and HCI
  • User-centric vs system centric
  • Emphasizes empirical vs theoretical work
Common themes:
Understanding the user, domain, and task contexts

Her Background
  • PhD in Mathematics and cognitive psychology (vision, perception, and cognition).
  • Joined the HCI group at Bell Labs in 1979
  • Introduction to IR, 1980-82
  • Human factors in database access
  • Vocabulary
  • Semantic Indexing
From Verbal Disagreement to LSI
They observed a mismatch between the way people want to retrieve information from a computer and the way that system designers describe that information.

Linux Command names
The trouble with Unix: cryptic command names (cat, grep, ls, etc…). It stimulated thought on the vocabulary mismatch problem.

They studied how people describe objects and operations in text editors, common objects.

She just had us pick names for a Boston weekend activity site/service and compare with our neighbors. Very few people agreed!

Findings: the tremendous diversity in the name that people use to describe the same objects and actions.

Details on alternative aliases:
Single keyword - .07-0.18 “repeat rate”
Single normative keyword: 0.16-0.35
Three aliases: 0.38-0.67
Synonyms for mismatch -- very funny!

Chi 1982 - 0th CHI Conference. Statistical Semantics: How can a computer use what people name things to guess what things people mean when they name things?
"In describing items in a database, however, system designers are at a disadvantage in that they do not usually get explicit, immediate, and continuous feedback from users. Knowing how people describe common objects and shift their descriptions from audiences of different levels of sophistication may help designers build systems whose information is accessible to the widest possible audience."
She's talking about earlier collaborators on LSI.

Tackling word mismatch

Rich Aliasing
  • Allow alternative words for the same item.
  • "Natural" in the world of full-text indexing, but less so for keyword indexing or command naming.
Adaptive Indexing
  • Associated (failed) user queries to destination objects
  • Add these queries as new document field
  • Quickly reduces failure rate for common requests/tasks
Latent Semantic Indexing
  • Model relationships among words, using dimension reduction
  • Especially useful when query and documents are short
  • Done earlier by Baker, Borko/Bernick, Ossario (1962-1966); Kohl (SIGIR 1879 p1). They lacked the computational resources to do this computationally.
Many applications and algorithms
- Bell labs directory of services, expert finding, reviewer assignment, handwritten notes, data evidence analysis, IR and Information filtering test collections

Rich aliasing and adaptive indexing in the web era
  • Full text indexing (rich aliases from authors)
  • Anchor text or tags
  • Historical query-click data (adaptive indexing with implicit measures)
"If you have an operational system and you don't use what your users are doing to improve, you should have your head examined"

The work on vocab mismatch started in CHI and built IR systems to help them, then the work went back to psychology.

More and more the work has been presented in an interdisciplinary fashion. Besides LSI, her most cited paper is in psychology:
A Solution to Plato's Problem: The Latent Semantic Analysis Theory of Acquisition, Induction and Representation of Knowledge

By 1995 or so, she was firmly entrenched in IR.

Common Themes
The last 10-20 years have been an amazing time to be in IR
  • TREC and evaluations starting in 1992
  • Search is everywhere - desktop, enterprise, web
  • Web search - big advances in scale, diversity of content and users, quality of results for some tasks, etc..
  • SIGIR community has a lot to be proud of, but many search tasks are still quite hard.
Common theme: represent and leverage richer contextual information about users, domains, and task environments in which search occurs.

Web Search at 15
  • Scale and diversity have exploded.
  • Number of pages indexed
  • 7/94 Lycos - 54,000 pages. (not the full text, only a few hundred words).. for IP reasons, not just technical reasons.
  • 1995: 10^6
  • 1997: 10^7
  • 1998: 10^8
  • 2001: 10^9
  • 2005: 10^10
  • (my note, 2009: 50^10+?)

Content Diversity
It started with Web pages and newsgroups. Now it includes maps videos, blogs, health... etc... the diversity of content has exploded.

How it's accessed
She showed the evolution (or lack thereof) the search interface across over a decade by comparing screenshots. It has not changed other than the width of the box (wider) and the label on the search button.
  • Has not changed as rapidly.
  • It's still a similar search box!
  • The result presentation has not changed significantly.
Support for Searchers
  • The search box
  • Spelling suggestions (getting the UI right is important to get right)
  • Auto-complete.
  • Inline answers
  • Richer snippets
Search in the future will look nothing like today's simple search engine interfaces said, Susan Dumais, adding, "if in 10 years we are still using a rectangular box and a list of results, I should be fired." (Mar 7, 2007)

In retrospect she regretted saying it, not expecting it to get picked up by the NY times. She got calls from Bill Gates and others telling here not to worry about it – she’d be retired by then!

Search Today
Query words --> Server --> ranked List
  • User context - where are the query words coming from?
  • Document context - Documents are related by interaction patterns, etc...
  • IR occurs in a larger task/use context -- search is done to solve a problem.
Search and Context (Sue's work)
You get context from: Users, Task, and Document Content.
  • Modeling users
  • Using User Models
  • Inter-relationships among Documents
  • Evaluation is very important
What problems are hard for people to solve? How can we build interfaces for people to solve them? Iteration and empirical testing is critical.

User Modeling
  • Modeling searcher's interests and activities over time (short-term vs. long-term, individual vs. group, and implicit vs. explicit)
  • Refinding and personalization

Refinding on the desktop
: Stuff I've Seen (SIS) (SIGIR 2003)
  • Unified access to many types of information
  • Index of content and metadata
  • Rich UI possibilities because it's your stuff and client applicaiton
Deployed six versions to over 4,000 people. They varied the query syntax, result previews, and the default ranking.

  • People issues shorter queries than web.
  • They don't use a lot of advanced search operators. filter, sorting
  • Date is by far the most common sort attribute (vs best match)
    - importance of time, people, episodes in human memory
    - few searches for "best match"; many other criteria
  • Need for "abstractions" in time - date, people, kind
Rich client-side interface
- support fast iteration/refinement
- Fast filter-sort-scroll vs next-next-next

Re-finding on the web (Teevan et al., SIGIR 2007)
  • 50-80 pages visits are re-vists
  • 30-50% of queries are re-finding queries
  • Total = 43% There is a big opportunity to support re-finding on the web.
  • There are few models to combine web rank w/ personal history of interaction.
  • Interfaces to support finding and re-finding.
  • Today: People et the same results, independent of current session, previous search history, etc...
  • PSearch (SIGIR 2005) : Uses a rich client-side model of a user to personalize the search results.
Building a user profile
  • Type of information: past queries, web pages, desktop
  • Behavior: visited pages, explicit feedback
  • Time frame: Short term, long term
  • Who: Individual, group

  • Where the profile resides:
    - Local: Richer profile, improved privacy (but, increasingly rich public data)
    - Server: Richer communities, portability
Using the user profile
  • Ranking
  • Query support
  • Result presentation
Ranking Algorithm (SIGIR 2007)

When To personalize? (SIGIR 2008, TOCHI)
  • Personalization works well for some queries, but not others
  • Models for predicting when to personalize using features the query and query-user
Evaluating Personalized Search (TOIS, TOCHI)
  • What's relevant for YOU, now
  • Explicit jdugments (offline and in situ)
  • Implicit "judments" from behavioral interaction
  • Linking explicit and implicit (e.g. curious browser study , 4000 volunteers. Used data to learn models that help you link implicit information from other people. If you use just the click, 45% of the time. 75% w/click+ dwell + session
Future Challenges

Dynamics and Search
  • As a community, SIGIR deals with static test collections (such as Gov2), but the Web is very dynamic, constantly changing. The community could use a dynamic information environment
  • Knowledge of how a page has changed over time is largely discarded by today’s search engines. Changes are important, and should be highlighted in snippets when applicable. For example, a SIGIR 2009 page was updated with information about a free breakfast: That would be really important for someone at the conference who was searching the website to help plan out their day.
  • Improved crawling policies
  • Improved ranking using temporal patterns, term longevity, etc.
Data and Evaluation
  • TREC and the Cranfield evaluation methodology is great, but don’t let it narrow the experiments we do. Search is inherently interactive and iterative.

  • User interaction data is important! How do users interact with existing systems, what are they trying to do, and what happens when they fail?
  • Implications for models and interactive system
  • Lemur query log toolbar – developing a community resource

  • The SIGIR community needs more large-scale log data. As a community, how can we get this? A community-based collection of queries and user interactions that can be used for research is needed.

  • Can the SIGIR community build a living laboratory where researchers can test ideas on an experimental platform? This would be an operational environment to test algorithms, do A/B testing, etc. (see the Microsoft Experimentation Platform)
General Challenge: Think outside the traditional IR box. What frustrates you? Understand users and their tasks. Solve their problems.

No comments:

Post a Comment