Thursday, July 22

SIGIR 2010 Industry Day: Lessons and Challenges from Product Search

Lessons and Challenges from Product Search
Daniel Rose, A9

Different Domains, Different Solutions
- Traditional IR,
- Enterprise search
- Web search
- Product Search
How are the issues different? Let's go back to user goals...

The Goals of Web Search
- Understsanding user goals in web search paper (WWW 2004).  Manually clustered queries until they were stable.
- Done at AltaVista in 2003 (not completely representative queries)
- Most product queries fell into other categories

Why do people search on Amazon?
- When they want to buy something?

Even ignoring the non-buying issues..

The Goals of the product Search
- Depends on where you are in the buying funnel.
-- Top: awareness, then Desire, then Interest, finally Action
St. Elmo Lewis, 1898
- Provide the right tools at the right stage in the process.

[roller coaster]
- toys and games
- sort by average customer review
- sort by price (is actually hard: new vs. used, amazon vs. third-party, etc...)

Different Tools for Different Stages
- Product search shows more fluid movement between searching and browsing behavior (relying on faceted metadata)
- Because of the nature of the search task?
- Because of the interfaces?

What Amazon Queries Look Like
- [which old testament book best represent the chronological structure]
- [shipping rates for amazon]
- [long black underbust corset] - still looking
- vs ISBN number -> about to buy it

(mostly one word, most the name of a thing.  except "generator")
top 10 across the us
(kindle, kindle fire, skyrim, mw3, sonic generations, cars 2)

queries in frequency deciles, by category
US, books, electronics, apparel
 --> very diverse, mispelling, miscategorization, all levels of the buying funnel

Context is King
- Some facets for Dresses vs. Digital Cameras
- The problem of facet selection
- Not a one size fits all UI solution for different facet types
- We can interpret your query in a smarter way: [timberland] boots inside shoes is a brand
- Timberland in music -> Timbaland the band (context dependent spelling correction)

Amazon is a MarketPlace...
- So search must be realtime
-- new products
-- new merchants
-- prices being changed all the time
-- items going in and out of stock all the time

Structured Data: "It's a gift... and a curse"
- Unlike the web search, we know the semantics of different bits of text
- We know what fields are important for customers (e.g. brand)
- A large degree of quality control (less adversarial problems)
- We don't have to do sentiment analysis to know if a review is positive/negative

A Curse
- Search engine needs to have both DBMS-like "right answer" behavior and IR-like "best answer" behavior
- Tradiontional IR mechanisms don't always work well for structured data
-- e.g. naive tf x idf  doesn't work well (see BM25F)

What happens when one of the fields is order of magnitudes bigger than others?
-- Search inside the book vs. brand name
- What happens when you don't have all the fields all the time? (missing data)
-- ratings, reviews correlate with user satisfaction, but it may not be there

Search Inside the Book
 - how often do you want to surface full-text matches vs. filter them out
 - (example query:  [byte-aligned compression])

Using Behavioral Data
- Powerful source of information for any search engine
- When is using behavioural data an invasion of privacy (or just plain creep), and when is it better for users?
- Customers of a business seem more comfortable with that business learning from past behavior.

Interpreting Behavioral Signals
Example: Are search result clicks good and bad?
- How many clicks are best?
-- 1: the customer found what their are looking for right away
-- many: comparison shopping and are looking around at multiple items
-- zero: the search result contained all the information necessary
Also, some items are inherently "click attractive", e.g. a book with a sexy cover

- "Why is the web so hard... to evaluate" (from snippet evaluation at Yahoo!) 2004

Evaluating Product Search Relevance
Common argument
-- Customers to to a shopping site to buy stuff
-- if a search engine change leads to customers buying mor stuff, they must have had their search need met more effectively.
-- Therefore, relevance can be measured by how much customers buy.
What's wrong with this argument?
-- besides ignoring the rest of the buying funnel, and that someone is ready to buy.

The A/B Test Mystery
- Compare ranking algorithms A and B
- Assign half of users A and half to B
- And the end the avg. revenue is higher in A than B.
-> algorithm A could be better than B, or Algorithm A could be recommending higher priced items than B
-> Algorithm A could be recommending completely unrelated, but very popular items.

So How to do Evaluation?
 - A/B tests, automated metrics, editorial relevance assessments (possibly crowdsourced).
 - Use all of them!

Lessons from IR
One idea: Generalizing the buying funnel
- The information seeking funnel
- Wandering: no information seeking goal in mind
- Exploring: have a general goal, but not a plan on achieving
-Seeking: have started to identify info needs that must be satisfied, but needs are open-ended
-Asking: have a very specific information need corresping to a closed class question
- Published in: The information seeking funnel, in Information-seeking support systems workshop 2008.

- Start thinking about how to meet user needs before user knows she has a need
- Offer different interaction mechanisms for different parts of the information seeking process
- Let type of content influence the way search works
- Design for realtime
- Interpret behavioral data carefully
- Exploit structure when have it
- Exploit context when you have it

(My Thoughts and Questions)
 - The world is not only Amazon.  What about linking the products to external sources, like consumer reports, dpreview and other sites?
  --> Amazon enhanced Wikipedia (e.g. Orson Scott Card)
 - Social, how is amazon incorporating social search?
 --> delicate balancing act with Facebook and other sources

 - Do you try and leverage mentions of products on book review sites? or within other books?
 - I recently went to barnes and noble and saw the new Orson Scott Card book, one of my favorite authors.  Why didn't Amazon surface that to me? (support for subscribing to authors)  Or, "buy the new top picks from this month's Cook's Illustrated"...
 - From my perspective, the recommendation quality of Amazon has decreased over time despite more of my data.  Does this reflect a shift in emphasis?

No comments:

Post a Comment