Sunday, August 28, 2011

The impact of online reviews: An annotated bibliography

A few weeks back, I received some questions about online consumer reviews, their impact on sales, and other related questions. At that point, I realized that while I had a good grasp of the technical literature within Computer Science venues, my grasp of the overall empirical literature within Marketing and Information Systems venues was rather shaky, so I had to do a better work in preparing a literature review.

So, I did whatever a self-respecting professor would do in such a situation: I asked my PhD student, Beibei Li, to compile a list of such papers, write a brief summary of each, and send me the list. She had passed her qualification exam by studying exactly this area, so she was the resident expert in the topic.

Beibei did not disappoint me. A few hours later I had a very good list of papers in my mailbox, together with the description. It was so good, that I thought that many other people would be interested in the list.

So, without further ado, I present you Beibei's annotated bibliography about online reviews and their business impact.

User behavior and online reviews

  • Nan Hu, Paul Pavlou and Jie Zhang, in their paper "Overcoming the J-shaped distribution of product reviews" have shown that the graphical representation of product reviews has a J-shaped distribution: mostly 5-star ratings, some 1-star ratings, and hardly any ratings in between. What can explain this distribution? They attribute this rating distribution into two biases:
    • Purchasing bias: People that buy a product do not constitute a random sample of the population. People buy products that they believe they will enjoy. So, the reviews are written by people that are more likely to like the product. Since only people with higher product valuations purchase a product, those with lower valuations are less likely to purchase the product, and they will not write a (negative) product review. Purchasing bias causes the positive skewness in the distribution of product reviews and inflates the average.
    • Underreporting bias: Among people who purchased a product, those with extreme ratings (5-star or 1-star) are more likely to express their views to “brag or moan” than those with moderate views.
  • Xinxin Li and Lorin Hitt, in their 2008 paper "Self-Selection and Information Role of Online Product Reviews" have found that online reviews may be subject to a self-selection bias: products are not randomly assigned to reviewers. Rather, early buyers (buyers who also post the first reviews) self-select product that they believe they may enjoy, in the absence of any existing information. This is in contrast to other buyers that wait for more signals about the quality of a product to emerge, before being convinced to buy, and therefore have a lower prior expectation about the product quality. As a consequence, the preferences of early buyers systematically differ from the broader consumer population, the early reviews can be biased, either in a positive or negative way. Such bias in reviews will affect sales and reduce consumer surplus, even if all reviews are truthful. 
  • Wendy W. Moe and Michael Trusov in their paper "Measuring the Value of Social Dynamics in Online Product Ratings Forums", looked into how social influences affect the subsequent ratings and sales. They demonstrated that reviewer rating behavior is significantly affected by previous ratings. In other words, product reviews not only reflect the customers' experience with the product, but they also affect the ratings of later reviews as well. 
  • Chrysanthos Dellarocas, Guodong (Gordon) Gao, and Ritu Narayan in their paper "Are Consumers More Likely to Contribute Online Reviews for Hit or Niche Products?" show that consumers tend to prefer posting reviews for obscure movies but also for hit movies that have already a large number of online reviews. The recommendation of the authors to owners of review websites is that volume of previously posted reviews should become less prevalent in order to encourage posting of reviews for lesser-known products.
Online product reviews and product sales

  • Judy Chevalier and Dina Mayzlin, in their 2006 paper "The Effect of Word of Mouth on Sales: Online Book Reviews" have first demonstrated that online ratings have significant impact on book sales. The key trick was to monitor the sales of the same book in parallel on and on Barnes & Noble. Since the two sites were selling the same book, any external effect would be similar to both websites. However, effects specific to Amazon or on would influence sales only on the respective websites (e.g., customer preferences on Amazon, site-specific promotions, etc.). Through this "differences in differences" method, Chevalier and Mayzlin could isolate and measure the effect of product reviews, without worrying about other confounding factors. 
  • Yong Liu, in the 2006 paper "Word of Mouth for Movies: Its Dynamics and Impact on Box Office Revenue" have looked at the same topic, but focused on the movie box office. Different from Chevalier and Mayzlin, his finding suggested that the valence of reviews does not matter for box office sales, however the review volume does. 
  • Pradeep K. Chintagunta, Shyam Gopinath and Sriram Venkataraman, in their 2010 paper "The Effects of Online User Reviews on Movie Box Office Performance: Accounting for Sequential Rollout and Aggregation Across Local Markets" have further studied the impact (valence, volume, and variance) of online reviews by looking at the local geographic movie box office, rather than the national-level aggregate box office performance. After accounting for various potential complications in the analysis, they suggested that it is the valence that seems to matter and not the volume. 
  • Jonah Berger, Alan T. Sorensen and Scott J. Rasmussen, in their 2010 paper "Positive Effects of Negative Publicity: When Negative Reviews Increase Sales" found that negative reviews can boost sales for unknown books, but hurt sales for books with established authors. This happens because negative reviews bring visibility to unknown books. Whereas for authors who are already well known, publicity does not boost the awareness of their books, instead, the valence of the publicity becomes more important. 
  • Chris Forman, Anindya Ghose and Batia Wiesenfeld, in their 2008 paper "Examining the Relationship Between Reviews and Sales: The Role of Reviewer Identity Disclosure in Electronic Markets" have looked at the role of reviewer identity disclosure (e.g., real name and location of the reviewer) in examining the relationship between Amazon book reviews and sales. They found that the prevalence of reviewer disclosure of identity information is associated with increases in helpfulness rating of the review and the subsequent online product sales. This is because community members more positively assess reviewers who disclose identity-descriptive information, and then use their assessment of reviewers as a heuristic shaping their evaluation of the product reviewed. 
  • Nikolay Archak, Anindya Ghose and Panagiotis G. Ipeirotis (yours truly), in the 2011 paper "Deriving the Pricing Power of Product Features by Mining Consumer Reviews", examine the idea that the textual content of the product reviews is an important determinant of consumers' choices, over and above the valence and volume of reviews. Using text mining tools, they incorporated review text by decomposing textual reviews into segments describing different product features. This work demonstrates how textual data can be used to learn consumers' relative preferences for different product features and also how text can be used for predictive modeling of future changes in sales. 
  • Anindya Ghose and Panagiotis G. Ipeirotis (yours truly, again), in the 2011 paper "Estimating the Helpfulness and Economic Impact of Product Reviews: Mining Text and Reviewer Characteristics", explored online review's impact on helpfulness and product sales, using multiple aspects of review text, such as subjectivity levels, various measures of readability and extent of spelling errors. The analysis has revealed that the extent of subjectivity, informativeness, readability, and linguistic correctness in reviews matters in influencing sales and perceived usefulness. See also the related blog post that I wrote in January 2010 (yes, even after acceptance, it took 1.5 years for the paper to appear in print). 
  • Yubo Chen, Qi Wang and Jinhong Xie, in their paper "Online Social Interactions: A Natural Experiment on Word of Mouth Versus Observational Learning" studied how word-of-mouth (WOM, i.e., others’ opinions) differs from observational learning (i.e., others’ purchase actions) in influencing sales. They have found that :
    • negative WOM is more influential than positive WOM;
    • positive observational learning information significantly increases sales but negative one has no effect (e.g., reporting purchase statistics help popular products, without hurting niche ones);
    • the sales impact of observational learning increases with WOM volume
  • Michael Luca, in his "job market paper" "Reviews, Reputation, and Revenue: The Case of" used a nice trick for estimating the causal effect of consumer reviews from on restaurant demand. Using revenue data from the state of Washington, he examined what is the effect of having an extra "half star" in Yelp. The key trick is to exploit the discontinuity in the way that Yelp assigns aggregate scores: A restaurant with 3.76 average review rating gets a 4-star review, while a restaurant with 3.74 average review rating gets a 3.5-star review. So, if there is a big gap in the revenues between restaurants with scores of 3.76 and 3.74, then this revenue gap (which actually exists) can be attributed to Yelp, and to its summary rating. (This blog posts presents further analysis of the paper, and also mentions similar use of this discontinuity trick to study the effect of sanitary scores in NYC: a restaurant may get an "A" score with $x$ penalty points, and another get a "B" with $x+1$ penalty points). Luca found discontinuous jumps in restaurant sales that follow the discontinuous jumps in the ratings around the rounding thresholds. This finding strongly suggested that changes in ratings (e.g., from just below a rounding threshold to just above a rounding threshold) can have significant causal impact on restaurant demand. 

 Online word of mouth and firms

  • Michael Trusov, Randolph E. Bucklin, and Koen Pauwels in their 2009 paper "Effects of Word-of-Mouth Versus Traditional Marketing: Findings from an Internet Social Networking Site" compared the effects of word-of-mouth marketing versus traditional marketing, as judged from the member growth at an Internet social networking site. They found that WOM referrals (i.e., invitations) not only produce a substantially higher short-term response, but also have substantially longer carryover effects in the long run than traditional marketing actions (e.g., promotion events, media appearances).
  •  David Godes and Dina Mayzlin, in their 2009 paper "Firm-Created Word-of-Mouth Communication: Evidence from a Field Test" examined how a firm should try to create useful word-of-mouth. They looked at who creates WOM and what kind WOM and matters. They found that for a product with a low initial awareness level, WOM that is most effective at driving sales is created by less loyal (not highly loyal) customers and occurs between acquaintances (not friends). They also found that although "opinion leadership" is useful in identifying potentially effective spreaders of WOM among very loyal customers, it is less useful for the sample of less loyal customers. 
  • Jackie Y. Luan and Scott Neslin, in their paper "The Development and Impact of Consumer Word of Mouth in New Product Diffusion" focused on how word-of-mouth (WOM) influences new product adoption in the video game market. Specifically, they were able to measure how effectively firms' marketing efforts generate WOM (buzz) and to determine whether WOM influences product adoption primarily through an informative role (i.e., helping the consumer learn product quality) or a persuasive role (i.e., exerting a direct impact on sales, for example, by increasing awareness).

If you have any other papers that you think that should be included in the list, please add your recommendation in the comments, together with a brief description of the conceptual and methodological contribution of the paper.