Michael Luca shows just how much restaurant reviews on Yelp affect companies' bottom lines. The more difficult question: Are these ratings reliable as a measure of product quality?
In recent years, consumer review sites including Yelp, Citysearch, and TripAdvisor have become the first stop for recommendations on everything from dinner to dentists. Along the way, they've earned a loyal following from fans, but also the ire of businesses that find themselves hurt by dyspeptic reviews.
“Restaurants are a classic example in economics where the consumer has to make a decision based on very little information”
Most would agree these sites do influence consumers' decisions. In the paper Reviews, Reputation, and Revenue: The Case of Yelp.com, Harvard Business School Assistant Professor Michael Luca set to find out exactly by how much, and identify winners and losers in the process. "I have always been interested in how companies form their reputations, not only restaurants and hotels but also schools and doctors," says Luca.
How reputation is earned in the digital age is a fertile area of his research. Luca is looking at rankings, expert evaluations, online consumer reviews, and quality disclosure laws to see how they work in market settings, and which are most important for consumers.
To get a handle on the phenomenon, Luca focused on one of the most common uses of online ratings: restaurants. "Restaurants are a classic example in economics where the consumer has to make a decision based on very little information," he says.
In theory, ratings sites fill in the gaps by providing a tremendous amount of information on which to base decisions. In Seattle alone, Yelp, which launched in 2005, had accumulated 60,000 reviews by 2009, rating 70 percent of the city's restaurants. By contrast, The Seattle Times had reviewed roughly 5 percent. Luca chose to investigate Yelp not only because it was the first restaurant site to emphasize user reviews over professional critics, but also because of its historical database that tracks every review.
Luca compared the ratings over time with revenue data from the state of Washington to gauge how reviews impacted restaurants' bottom lines. In order to establish causality between reviews and a restaurant's success, Luca took advantage of the fact that Yelp rounds its ratings to the nearest half-star-so, for example, a 3.26 rating would be rounded up to 3.5 stars, and a 3.24 rating would be rounded down to 3 stars. Hence restaurants with similar ratings can have very different rounded ratings displayed to consumers. By applying a regression discontinuity analysis, Luca could study how revenues jumped when star thresholds were crossed and compare these with the more steady changes over time. The result: Luca could directly trace the effect to the ratings on Yelp.
When the dust settled, Luca determined that each ratings star added on a Yelp review translated to anywhere from a 5 percent to 9 percent effect on revenues (depending on the control variables and means of estimation)—more than he had expected.
Even more interestingly, within that number not all restaurants were created equal. Chain restaurants, in particular, were largely unaffected by the ratings, while the greatest effect was shown for independent restaurants. That makes sense according to economic theory, says Luca, since diners presumably already have some knowledge about chain restaurants, but can benefit from more information about their neighborhood spots.
"Yelp is somewhat of a substitute for traditional forms of reputation," says Luca. "People are not using Yelp to find out about McDonald's."
But Luca says the big chains should not be comforted by the findings. Indeed, he believes the data suggests that local eateries are starting to siphon off customers from the "Big Boys." Why? The Applebee's and the T.G.I. Friday's of the world have been safe bets for diners because their fare is of an expected quality and their menu well-known—thanks in part to big-budget advertising behind the chains. But review sites are leveling the playing field by allowing consumers to learn as much about independent restaurants as they know about the chains. "This is one reason why consumer demand is shifting from chain to independent restaurants in the period following the introduction of Yelp," Luca writes.
The more difficult question to answer is whether the public's reviews are reliable measures of a restaurant's actual quality. Restaurant owners routinely grouse about being held hostage to their Yelp ratings, accusing other restaurants of gaming the system by stacking reviews by friends and family, or simply falling prey to a popularity contest by uninformed palates.
"Given the evidence, I don't think many people would dispute Yelp's influence," says Luca. "The question is, is this a good thing or a bad thing? If the ratings are capturing real quality, then that's a force for good. To the extent that there is gaming or nonrepresentative views, that's a problem. The onus is on the review industry to change that."
“If the ratings are capturing real quality, then that's a force for good”
Luca has been following up his initial research to determine how review sites can do just that. He notes that consistency of reviews were weakest when there were a low number—supporting the criticism by those restaurants that blame Yelp for killing business with a couple of bad reviews. "Some of the complaints I hear are from restaurant owners in small towns who get one or two reviews—it has an outsized influence on your business. It can be hard for them to wait for more reviews. Once you start getting 20 or 30 reviews, the variance goes down," says Luca. The flip side, of course, is that most small restaurants in smaller communities wouldn't get reviews to begin with if it weren't for the review sites, and for every eatery that suffers under the weight of a bad review there are presumably others boosted by positive reviews.
Luca applauds features on Yelp that try to improve content quality, such as requiring reviewers to create publicly available profiles, and identifying some as "elite reviewers" based on their number of reviews, so users can give them extra weight. Holding users accountable with social networking tools makes Yelp stronger than similar sites like TripAdvisor, which doesn't have such accountability. On the other hand, Luca approves of the more extensive options that TripAdvisor provides for sorting and categorizing reviews of hotels—for example, business travelers can find reviews written by other road warriors, who are likely to have similar requirements.
"The ability to quickly find reviews from people with similar tastes is one potential area of improvement for review systems," says Luca. "At this point, there is no single site that has mastered it all."
But even as consumer reviews influence more and more purchase decisions, websites must be concerned with the larger picture of all the factors that create trust between them and their users. "Reviews are one piece of the reputation puzzle, but a comprehensive reputation system must do more to facilitate trust by incorporating other types of information as well. For example, Yelp could include other publicly available information about restaurants."
Expanding on this research, Luca is working with HBS Assistant Professor Benjamin Edelman to put a framework around reputation-building in online marketplaces. They are currently finishing work on a case study about Airbnb, where anyone can offer a room in their house or an apartment for short-term rentals.
"Compared to restaurant choice, people are taking a big risk when deciding to rent their apartment to a total stranger or to stay in a stranger's house," Luca says. Because of this, Airbnb offers not only a review system but also insurance to pay for incidents resulting from unreliable guests, authentication to make sure that guests and hosts are who they say they are, and a screening process to give market participants the ability to decide when and when not to do business with each other.
"The right mix of these tools is very context-specific, and this case helps to highlight some of the challenges to building a robust reputation system," Luca says.
As it becomes more de rigueur for consumers to use website reviews and other information to make important decisions—from where we are going to vacation to who will treat us when we're sick—it's all the more critical that businesses provide consumers with enough quantitative and qualitative information to ease their decision making.
And that has Luca thinking. "How can we extend this to other industries, and how can we better design these sites?" says Luca. "It really comes down to what types of truths we want to find in each site and what we can do to ensure reviews are of a high quality."
It is similar to Amazon reviews, when people were astonished to see negative reviews about the books and were wondering why Amazon is publishing them. This definitely translated into increased sales for good books overall benefiting Amazon. Am not sure if Yelp has some way that the restaurant owners can get in touch with negative reviewers and give them coupons to asking them visit again to see the improvements done. Or if they have a way to reach out the positive reviewers and give them a free drink the next time they show up !
Not sure I'd rely on a bunch of unknown goobers' opinions when choosing a company with which to do business or a place to eat.
Too many favorable review and it's all the owner's pals. Too many negitive reviews and it's the competitor down the street and all his pals.
Use a website to find someone to treat us when we're sick!! Oh yeah, right.
Interesting. I wonder if Yelp has advanced thinking around 1. aging comments out of the scoring to recognize that services levels are always changing (ie. bad comment from 6 mos holds less weight than one from last week) or 2. allowing for scoring improvements if the Store responds with a positive solution to a bad score, or 3. analyzes other factors such as local customer, repeat customer, vs a 1 off customer who isn't local.
The author is mistaken.
Zagats pioneered user-based reviews at least a decade prior to the inception of the Internet. Pre-Internet they controlled quality by developing relationships with consumers who frequently ate out.
Yelp may have extended this concept more broadly but they certainly were not pioneers.
Perhaps is why Google acquired Zagats in September.
Looks like a classic example of how statistics can be misused. The author correlates Yelp ratings with revenues earned by restaurants and concludes that changes in ratings have caused changes in revenue. The reality may well be that the restaurant is serving better food and consequently the revenues are improving and , obviously, the ratings.
What do you think?
We recognized the fallacies of review sites such as Yelp and have sought to provide an alternative for consumers which is based on reviews and opinions from their social networks. Cliqsearch.com allows you to search your social networks to see if you've had friends or friends of friends who have been to a particular restaurant or used a particular service and so on. Based on your knowledge of that person and your relationship to them, you can make a better informed decision.
B2Beware: While consumer ratings are extremely useful, similar ratings for B2B products and services may suffer from gamesmanship, especially when the market is small and the buyer set competitive. For example, a large expert network discovered that its users' subjective ratings were inversely correlated to their repeat usage behavior. Users evidently wanted to keep for themselves the best experts--preferring to gain a competitive edge over helping the experts and other users. The expert network solved the problem by changing its rating system to one based directly on repeat usage behavior.
how can we better design these sites?
We could have better attractive reviews ,with classifiable rating method ,mixed online relation with new costumer and who write reviews for the best decision making.
A review is that. a review.
The issue with some forms of social media that no standards are enforced. It becomes subjective. not objective. I know nothing about accounting; but I'll give a performance review on . you get the drift. You can critique the meal etc. but don't grind an axe or get on your "I know it all soapbox". some folks should not be allowed to eat out; let alone use a drive through. I also believe it you place a review..you should have to post a picture and contact information. If you are going to help/comment/hurt or . to any business. You should have to own it.
I have been in buisness over thirtry years and although social media can/is a great tool; it needs to have standards too. Ever try getting in contact with a real person on a some of these sights ?
You have missed one main data. Yelp, Citysearch frequently calls restaurant for package deals. Once you take the deal, the ratings go high. They may be employing people to add 5 star reviews. As a restaurant owner, paying 200$ a month to get 5 or 10 extra 5 star reviews is not possible. Once you are off the package, they add the 1 star reviews. This is not discussed openly, and customers are not aware of it.
I am very interested in the last comment on yelp/city search actual contribution to positive/negative ratings.
One business model I am looking at addresses the "fake reviews" by only allowing those who have purchased (say a haircut) to provide reviews via email in exchange for a small monetary incentive/discount. The salon makes the review option available to all customers upon check in, a third party company contacts them later via email for the review and reward.
I'd love to hear people's thoughts.
Cosumer review sites are serving a good purpose. Through these sites we get insight into consumer behavioral patterns. But chances of subjectivity in arriving at the findings cannot be ruled out if the sites call for information on questionnaires where answers to queries are got rated say from 0 to10. Big range and high probability of people answering whimsically and not logically.
A further intelligent handling of the results is necessary by avoiding blind faith.
A while back I would have been delighted with consumer review sites, lately I am have major issues with TripAdvisor to the point that I have been totally turned of the site and will never use it again.
I do not know if you have seen the documentary following the people who call themselves "professional reviewers" and go around restaurants or guest houses looking for problems just to post a negative reviews and gain a following, horrific and proof that some review sites are a joke and to be avoided.
Most of you have great comments are commended for caring enough to comment. Reviews do matter and as a consumer, i'd rather be armed with some information about a business I'm about to engage than not.
How do you know if you can trust the reviews? The answer is RaveOrBash. Check us out when you have a chance. We are launching first quarter of 2013.
The next level in consumer reviews arrives next year.