Ratings are Overrated

I’ve been thinking about this post for a while now, and something that happened recently hit home hard enough to really send me over the edge.

So let me go ahead and put it out there:  I have a problem with user ratings.

I have been a member of Amazon for 10 years.  I am completely loyal to them and probably spend a minimum of $1,000 per year with them.  I have had similar relationships with other sites that have soured, while Amazon continues to stand tall (no pun intended).

One of my reasons for being so loyal to Amazon is that their member rating system is beyond compare.  When you read user reviews of books for example, you are pretty much looking at the real deal.  For the most part, people say what they think.

Sure, there are authors who will try to artificially increase a particular book’s rating by having all their friends and neighbors and parents and fans go on to the site and give it good ratings.  However, I’d like to believe the law of averages will eventually take care of that and besides, most readers are savvy enough to see through these games.

Now, contrast this with another site, (which I won’t mention but if you email me I’ll tell you privately), that lauds itself in its marketing materials because “our vendors get all 4.5 star or above ratings.”

Can someone explain to me how this is useful?  What does it say about this company?  That any vendor below 4.5 stars gets summarily kicked off their site?

NO!  What it says is that they “encourage” their raters to give good ratings.  So please, tell me how this helps me, as a consumer, to decide whether to use this service or not?  In other words, if everyone gets 5 stars, what is the point???

It reminds me of the 60-Minutes article a few years about about the Millenials, who feel that everyone in the game deserves a trophy whether they won or not.  Which is ridiculous when you think about it.  A game isn’t a game if someone doesn’t win.  Giving trophies to the losers sets these kids up for a lifetime of unreasonable expectations.

In addition, how does the losing team (or poorly performing vendor) ever learn of their weaknesses, in order to try to improve them?  Not only is it not fair to the readers, it’s not fair to the vendors!

Here’s another example:  A couple of months ago I ordered two exercise tapes from a seller on a popular auction site.  The condition was listed as “Like New” which to me means the box is open and maybe the tape has been played once or twice but it’s otherwise indistinguishable from a new item.

I received the tapes and after watching them both, found that one of them was in only fair condition.  The color was off, the viewing was scratchy, and the sound was inconsistent, which is indicative of a VHS tape that has either been played too many times or sat in someone’s trunk for a half a year.

The tapes were cheap and I did not want to send them back, I was happy enough with  my purchase, but I wanted to make a point.  So I gave the seller a “neutral” rating and stated that one of the tapes was not in the advertised condition.

An interesting thing happened.  The auction site REALLY did not want me to leave this less than perfect rating.  It discouraged me *strongly* and made me agree to a list of statements by checking off a series of boxes, before it would finalize my rating.  Huh???

Now, for the icing on the cake… In last week’s post I mentioned that I had hired two VAs.  One of them delivered substandard work, and I felt that I paid her for 10 hours to do something I could have done in about two.  So I politely told her I didn’t need her any more and ended her assignment, and generously gave her a 4-star rating.  She had all 5-star ratings previously, which honestly stumped me a bit given the poor quality of the work I got from her.

She emailed me and asked me to please change my rating to all five stars, because she “cares about her reputation.”  She mentioned that she had given me five stars in return and so would I please reconsider my rating.  Well, I did not respond to her because what I really wanted to tell her was that she was lucky I didn’t give her two stars.

Whatever happened to “Gee why didn’t you like my work? What could I have done to make you happy? Can I make it up to you?”

I have had several of these WTF moments over the past several months regarding ratings.  I have thought a lot about it, and I have come up with a list of possible reasons for why this trend may be taking place.  These really are guesses, I claim no expertise in this field except for a healthy dose of insight into human nature.

  • People don’t like conflict and giving someone a poor rating “to their face” is hard to do.  Amazon’s products are very impersonal, which makes it easier to be honest.  Hiring a freelancer to do work for you on the other hand, is very personal.
  • Most people don’t know how to give constructive criticism.
  • Sites that support ratings systems have come to believe for some reason that having better overall ratings makes their site or product more desirable.
  • People would rather not give a rating at all if they have a bad experience, which skews the ratings (but they WILL tell their friends about it).
  • People believe that they deserve a good rating automatically, without having to earn it.
  • Somewhat related to the previous point, but worth its own mention is the whole concept of the entitlement generation.  Urban legends about millenials whose mothers call their bosses when they don’t get a favorable annual review come to mind.

I’d love to hear what others think of this, whether you agree or disagree.  If you agree, can you offer your own suggestions for what might be happening here?  If you disagree with me, why?  Really, I want to know.

* * *

Do you enjoy reading these posts? Why not sign-up to receive Grouvia’s e-newsletter? You’ll get the latest news delivered to your inbox and you can participate in the Grouvia development process. It’s free. Sign up at www.grouvia.com.


7 Responses to “Ratings are Overrated”

  1. Susan Weinschenk Says:

    I give this blog 5 stars (ha ha). Seriously, these are interesting points… People seem to behave differently when they are rating someone they know or have a personal interaction with than when they are rating a product or author or someone they don’t know. In the latter case I think they are willing to be more honest/negative. Although you raise issues about the validity of ratings, i have to say that ratings are here to stay. It’s the principle of social validation that I write about in my book Neuro Web Design: What makes them click? People want to know what other people say, and what other people say influences them whether they want it to or not…

  2. Lisa Pecunia Says:

    Susan, thanks for your commments. Yes, I absolutely agree that ratings are here to stay. And for good reason, as you stated.

    I think used well, peer reviews and rating systems are highly useful, for the reason you mentioned. But like any form of influence, marketers will try to exploit it to their advantage, and the marketers who are not as ethical as the rest of us may do damage to it in the long run.

  3. uberVU - social comments Says:

    Social comments and analytics for this post…

    This post was mentioned on Twitter by karls_mlab: Great blog post: The Making of Grouvia: Why Ratings are Overrated http://bit.ly/3a0tAt

  4. Twitter Trackbacks for Ratings are Overrated « The Making of Grouvia [grouvia.com] on Topsy.com Says:

    […] Ratings are Overrated « The Making of Grouvia making.grouvia.com/2009/10/22/ratings-are-overrated – view page – cached I’ve been thinking about this post for a while now, and something that happened recently hit home hard enough to really send me over the edge. — From the page […]

  5. uberVU - social comments Says:

    Social comments and analytics for this post…

    This post was mentioned on Twitter by dana_clark: How useful are user ratings? http://bit.ly/25jj9w

  6. Paul Bryan Says:

    Sorry for the long reply, but I care a lot about this topic.

    As an overall system, I think that the ratings disparity will correct itself. Here’s why.

    Two complex variables implicit in the ratings system are Trust and Authenticity. They are not independent. Trust is a characteristic of the people viewing the ratings. Authenticity is a characteristic of the people responsible for the ratings.

    In user research sessions, I have met some people who tend to trust a source until it is proven untrustworthy. I have met others tend to distrust a source until it is proven trustworthy, particularly when the source has a vested interest in a positive rating (ok, I’m in the latter group). Both groups will lower their trust for a given source when they have an experience that belies that trust. For example, you trusted the rating that your VA had received in the rating system, and were disappointed with the results you experienced. There could be many reasons for this disparity. But in the end, you put less trust in that system because it had betrayed your initial trust level. You may try to correct that system by assigning a slightly lower rating, but in the future you will definitely have less trust in that system’s ratings.

    On the other side of the ratings system, the people supplying the ratings practice a certain level of authenticity (the quality or condition of being trustworthy or genuine). Some companies supply the ratings as they are submitted. Amazon doesn’t need to care if people don’t like the book. They can find another one. An online shoe retailer, on the other hand, has to care if somebody says a certain shoe brand is crap. That may affect a lot of purchases, particularly if the customer repeats this assertion in a number of places online. The shoe retailer has to decide to what degree they should “adjust” the ratings by omitting negative outliers. Rating by rating, they are building the perceptions of their authenticity.

    The authenticity of a given source of ratings is not known until an experience reveals it. You had expected a degree of authenticity from the VA ratings, because a lack of authenticity defeats the purpose of the system. The people supplying the ratings were (apparently) not authentic. Their lack of authenticity was discovered through experience and you adjusted your trust level. Others probably have had the same experience. Together, a large group is lowering its trust, and the perception of authenticity is decreasing overall.

    Companies used to be able to control the conversation that impacted the perception of authenticity, thus engendering trust by spending large amounts of advertising dollars. But not anymore. That conversation is out of their hands. In aggregate people will continually adjust their trust level, and communicate those adjustments to others, which will cause harm to the reputation of authenticity of a given source. In user research sessions I have found that lack of trust is one of the most difficult barriers to overcome in a transaction-based web site, so a decrease in aggregate trust will be felt in the bottom line.

    This bottom line impact will take a while to surface. It will eventually be recognized as a problem by the companies that lack authenticity in their ratings. But by then it will be as difficult to recover trust in the ratings system as it is in every other context.

    Paul Bryan
    Usography ( http://www.usography.com )
    Blog: Virtual Floorspace ( http://www.virtualfloorspace.com )

  7. Lisa Pecunia Says:

    Paul –

    Thank you for this excellent analysis, it’s clear the topic is important to you. Although I could never have put it as eloquently as you, I completely agree with your final thoughts.

    Those companies (or sites or apps) that do have a problematic rating system will find out eventually. Even if the problem is not due to a lack of authenticity in the organization as a whole. I think these issues are in fact invisible to most people, as well as the companies themselves.

    Both the company and the users get hurt by this, as we can see in the examples described.

    Personally I have chosen not to implement a rating system on my site (Grouvia.com) for this very reason. I will wait until the day when some innovative technologists find a way to create a truly transparent “recommendation engine”, and then I will implement that. It’s probably a pipe dream for now.

    Thanks again for your truly insightful comments.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: