A member of the TTAC B&B wrote in to Robert this morning with some unhappy feelings. He feels that stars for cars are going out like free money in the recent months at The Truth About Cars. In this reader’s words:
I had no idea that writers’ star ratings are not edited. Are writers given any instructions on how to rate? In keeping with TTAC’s take no prisoners reputation, I would like to see some uniformity in the ratings where
Top 5% get 5 stars
Next 15% get 4 stars
Middle 60% get 3 stars
Next 15% get 2 stars
Bottom 5% get 1star
In other words, I want to see most cars get a 3-star rating and, as a corollary, a 4-star or 5-star rating to really mean something. This would be really helpful to car shoppers. Currently, however, your writers seem to be giving 4-star ratings out like hot cakes.
Well here’s your chance, B&B. Tell us if you think we are experiencing star inflation. Obviously, our usual rules about flaming the site are suspended. But keep it civil, or I’m going to bust out the whoopin’ stick.
In certain cases, definitely. To borrow from what I wrote in the VW Tiguan review:
“Because of the inherent drawbacks of CUVs as a class, they should have to be absolutely exceptional to warrant a 4-star review. The entire class only exists because people no longer want to drive station wagons.
While the Tiguan may be superior to a Rav-4 or CRV, how does it make sense with cars like the GTI or Jetta Wagon around? It’s more expensive, handles worse, slower, not much more cargo room, uses much more gas, the interior doesn’t look that great from the pics I’ve seen, etc.”
Vehicles that don’t even make sense like the Tiguan, Nissan Rogue and BMW X6 have all gotten (I think) 4 or 5 star reviews. And this is with alternatives available on the market that do nearly everything better, while being cheaper as well.
NO! It’s fine as is. This is The Truth About Cars. Let the opinions reflect the qualities (or lack thereof) of the cars as tested.
Grading on a curve is for idiots, and only reinforces “expectations” of lowered expectations.
You asked. That’s my opnion.
There do seem to be a lot of four-star reviews. This is probably fair; most new cars are pretty nice if not stellar.
Trying to summarize a well-written review into an arbitrary number seems dishonest and only detracts from the author’s efforts. Let the article speak for itself.
There are star ratings? I never noticed.
On a serious note, modelling this after Consumer Reports might be a good idea. Their performance ratings are largely quantitative where possible and easy to compare. Stars and scores (1-10, percent, etc) are more difficult, trying to be both quantitative and qualitative and not doing either well
This begs the obvious question – Does anyone actually believe the “best and brightest” rely on a star rating to select a vehicle, or rather, is this system seen as a benefit for the “worst and dimmest”?
The star ratings do seem a little generous. I’d rely on the text and comments and a thorough test drive instead.
I will start off by saying, an x out of y ranking is not the best way to review car, but at the same time I love to compare them.
I think Roger Ebert’s theory about assigning star ratings to movies would work well in assigning star ratings to cars. When asked how it was possible to give both The Godfather 4 stars and Spiderman 2 4 stars, Ebert explained that he doesn’t rate movies on an absolute scale. Spiderman 2 was not trying to be the same type of movie as Godfather. It’s a superhero movie, and as a superhero movie it’s a 4-star one.
I guess as a reviewer you need to determine what a movie/car is trying to be and then judge how successful it is in that endeavor. And if a movie/car is not clear in it’s purpose, chances are it’s not going to be too successful in fulfilling its potential.
And yes, there do seem to be too many 4 star ratings lately.
If anything I’d call it star review stagflation. The stars are going up while at the same time the new model emphasis is going bust (from the driver’s perspective).
Grading on a curve is for idiots, and only reinforces “expectations” of lowered expectations.
This begs the obvious question – Does anyone actually believe the “best and brightest” rely on a star rating to select a vehicle, or rather, is this system seen as a benefit for the “worst and dimmest”?
I disagree. The star system is useful; it provides the reader a quick lens by which to gauge the author’s overall impression of the test car.
As for grade inflation, TTAC shouldn’t be in the business of blowing sunshine up a car’s arse. If we don’t use 1 and 2 star ratings, you might as well not include a rating at all. A Gaussian distribution is definitely warrented.
Yes, and I’d eliminate the star ratings entirely. TTAC reviews are entirely subjective and supported by very little data. That’s OK, but it is what it is. TTAC reviewers evaluate cars like Absolute Sound evaluates high-end audio gear but with fewer words and less nuance.
For anyone who won’t take the time to read the review, the star ratings are content-free and uninformative of any qualitative measure. A five point ranking system is far too crude to communicate the on-balance summation of these subjective reviews, which are made all the more subjective by the widely-varying personal perspectives of TTAC reviewers and their inconsistent criteria for acceptance. So eliminate the stars. TTAC reviews are only useful to those who actually read them, though the spec summaries and other compiled information can be a good reference having nothing to do with the subjective evaluations. Beyond that, TTAC reviews are also entertainment.
Phil
Maybe the cars are just that good.
Mixed feelings on this. I think there’s some confusion as to whether the star rating represents the reviewer’s personal reaction or their perception of how well the vehicle scores compared to what it’s intended to be. For instance, Jonny might not like cushy luxo-mobiles any more than I do, but should a vehicle that’s clearly trying to be a luxury ride be penalized for not handling like a Lotus? That’s not always clear.
I wouldn’t mind a split star rating system. A rating for how well the vehicle succeeds in its mission, another for the reviewer’s feelings about it from the perspective of a knowledgeable enthusiast. There are plenty of perfectly competent vehicles out there that engender a “Why, God, why?” reaction from people like us.
I do think there have been some inflated ratings that seem based less on a qualitative judgment and more on the reviewer’s astonishment that the vehicle actually does something competently — for instance, a CUV that doesn’t handle like a three-legged steer.
I read the ratings below the stars because they are funny, and I obviously savor the text of each review, but generally completely ignore the stars. It’s just too difficult to judge. Might be useful for car shoppers stopping by TTAC (the original reason why stars came in), but it’s too subjective. Are we rating by class? In the grand scheme of things? How do each reviewer’s personal feelings come in the rating? And so on.
As far as I’m concerned, I know I’ll NEVER buy a CUV or SUV, no matter what its star rating is. And when I’ll go car shopping, I’ll make sure to read beyond a few stars at the bottom of a review. That just shows how subjective it is.
I’m with psarhjinian, but think the true superiority of CR’s ranking system is that the cars are ranked in their respective classes. I think the fur would fly, major debates be launched if cars were slotted in above and below one another, clear winners and losers would emerge by relating the cars to each other.
I don’t know what you do when a new TSX ends up being inferior to its predecessor, but leaving both in the rankings could also indicate a manufacturer’s current commitment to a car or segment.
I think that the star system should be like the Netflix rating system. Perhaps something like this:
5 stars: I love this car, and everything about it. I would give my right arm just to spend 1 hour driving this thing anywhere at anytime.
4 stars: I really like this car. It does what it sets out to do with excellent execution and design. I would be thrilled to drive this car on a daily basis.
3 stars: This car is okay. I could take it or leave it. It does what it sets out to do okay, but it does nothing to excite me.
2 stars: This car is not-okay. It fails short in several areas, and I do not want to drive it. Design and execution are lacking.
1 star: I would give my right arm NOT to drive this hunk of junk. What were the idiots thinking when they decided to try to get people to pay money for this pile of #*$#!
If you notice, this system is more on an emotional level than an objective level, but I think that works better than an objective approach for this site. Otherwise, the Camry and Accord would be 5 star cars, but would you give your right arm to drive them?
For me, grading an entire vehicle with a maximum of 5 stars is silly. Clearly, most cars will get 3 stars as most cars are average. That is, the majority of vehicles define what is average and thus MUST be average.
If you were serious about grading cars, leave the individual area ratings 5 stars but make the overall rating out of 10. For argument’s sake you could then say that a Malibu is 7 stars and the Aura is 6 even though they are both (let’s assume) considered average.
In addition, if you are going to have a group of people giving out subjective ratings, you need to establish a baseline. Something that everyone can either agree on or something that is defined. One example could be that a GTI would get the 5/5 for suspension.
Without standards, each writer is giving a subjective rating with a different baseline so your ratings really don’t make sense when compared with each other.
Without more stars, the system makes no sense because too many cars will have the same rating, removing the primary (desired) benefit to car shoppers.
Leave the star system as is. I don’t see how you could adopt a “gaussian” system when different writers review different cars in different locales. Certainly don’t make it more complicated.
Or putting it another way, how would TTAC decide what the “middle 60%” represent when not all cars of a given model year are reviewed at the same time over the same roads and by the entire TTAC group of writers?
I’ve disagreed with ratings of cars on TTAC but I don’t think they are overall guilty of inflation. They may be guilty of reviewing cars that appeal to them and so the subsequent ratings are high. I’ve read the past disagreement over an SUV rating where many thought the rating was way too high. I think the rating was justified when rating an SUV as an SUV. As an overall rating I think SUVs should get a half star but within the rating criteria, of rating it for what it is, I am wrong. That said I think some of the ratings of small inexpensive cars suffer from over-expectations. A cheap-ass economical commuter shouldn’t be reviewed for heel/toe shifting pedal placement. I thought the Forester XT rating was overly harsh and the three stars didn’t reflect the criticisms made in the text. Maybe that’s inflation. Would the base Forester garner 4 stars? If the XT was sooooo bad then why three stars? Three stars to me is average.
TTAC reviews are entirely subjective and supported by very little data. That’s OK, but it is what it is.
TTAC is author’s own opinion, without hard facts to backhold it. I have other sites to compare cars in authoritative and methodical, almost ordinant way.
In TTAC reviews I like to read between the lines and see what other reviewers are missing to spot out.
Many times I get annoyed by the reviewers subjective dispositions, but at the same time, this relieves them of any responsibility, making them more likely to be sincere in their reviews.
As for the stars system, I completely ignore it, because that’s not what I need to read there.
It seems cars here are marked either “good” (usually the fun-to-drive enthusiast oriented cars) or “bad” (opposite) with few “average” cars. That can be expected since the majority of the writers and readers of the site are auto enthusiasts and the majority of us would prefer our car to be more than just transportation and utility. That said, maybe you should consider the old fashioned percentage-based score. 5-stars just doesn’t give you enough info at a glance when comparing cars. Ex: Two cars may be rated at 2 stars. But one of them may be a 45% and the other a 65%. Basically it comes down to “How bad is bad and how good is good?”. The stars show them as being equal when they really are not, percentages give more accurate info for comparison.
One thing that might be useful to clarify is whether the vehicle is being ranked on a scale compared to other vehicles in its “class,” or all vehicles period.
To me, it would make sense to rank them just within some general class (deciding whether a Land Rover is “better” than a Boxster seems meaningless). But if we do that, then the class should also be identified next to the number of stars.
rating systems always seem to beg the question, compared to what. What I look for in a review, especially on this site, is how well the reviewer enjoyed driving the car and whether or not there were any serious drawbacks to the experience of driving or riding in the car.
What would have the most meaning would be a thumbs up or down. Thumbs up meaning I liked this car and would part with some of my earnings to buy one. Thumbs down meaning this is a car that I would not buy. so I guess there are really only two ratings that are important, one and five.
Something like a Lexus, which is derided for being a boring bed on wheels should not be penalized for being a boring bed on wheels, it is the BEST boring bed on wheels and should get 5 stars for that. The review can complain about how numb the entire experience is, but in the end, the stars should reflect that the car itself is the absolute best at doing what it set out to do. Just as, if you take out some ridicules exotic that breaks down 3 times in 3 hours of testing, but thrusts from naught to 60 in half a second, it still does exactly what it is out there to do. The reviewer can tear apart the car for its reliability (or lack thereof) but the stars should reflect where it sits in its niche. The inverse, of course, is the Pontiac Solstice (sans turbo), a sports car with no sports in it. A fine car, I’m sure, but you think to yourself “why buy this when you could have ___”, that should get a very low rating. Whenever I point people here, or anywhere else to read reviews, I always caution them about the bias. I tell them to read the entire review and find out what it is they like or don’t like and then, decide for yourself if that matters. If they complain that, while on the highway, it doesn’t downshift fast enough and you drive 90% of the time in the city…ask yourself ‘how much does that matter?’
@ Jonathan I. Locker
You know… that makes sense. Ultimately, the review comes down to a subjective opinion from the reviewer (with the text explaining the specifics), and this makes that clear and understandable. I also like the emphasis on “doing what it sets out to do,” rather than simply how close it is to a “perfect car,” whatever that might be.
Instead of stars, you should provide the relative standing of the vehicle w.r.t its closest competitors
For example, a VW tiguan review would be summarized as:
1. Rav4
2. Tiguan
3. CRV
4. Vue
or something like that.
Intuitively, it seems we have seen too many 4 stars recently. That likely generated the complaint.
I am not a big fan of overall ratings for cars myself. Too subjective, and not really informative.
Though I am not the B&B, I still gotta give out the $0.02.
In other words, I want to see most cars get a 3-star rating and, as a corollary, a 4-star or 5-star rating to really mean something.
I hated bell curve grading when I was in school, and I hate the idea here too. Some cars, like the Dodge Challenger, have so few comparable peers that it makes no sense to compare it to just the Mustang. Or the G8, if you subjectively think a coupe is similar to a sedan.
And my 5-star review of the Genesis 4.6 was because there is no 375hp, RWD, power everything, sedan at its price point.
Restricting the writer’s creative input here is a long and slippery slope…that (quite frankly) none of us have the time to deal with.
I’d forgo the stars. I hardly look at them…and only when people have referenced the rating do I scroll back up.
Maybe do a simple (relative and subjective) comparison:
Worse than…
Better than..
To be honest, I pay little attention to the star rating. I read the review and then decide for myself how I feel it ranks. But, to be honest, I am surprised at some of the 4-star ratings after reading rather harsh reviews.
I’ve never felt the points system is off, as a whole.
A high star rating should mean that the vehicle achieves , for example, 80% of it’s intended goal.
You can’t not give a Audi R8 5 stars because it gets lousy mileage and has poor cargo room, thats not it’s intention. I have never objected to any star ratings offered on a car, because that is just the reviewers opinion on the overall ‘success’ of the vehicle. How well it hits the mark.
One way to improve perhaps would be to have a larger range, maybe ten stars, and break it down based on different criteria.
Like Driving Performance (Does the sporty car drive like one?), Value (Competitive pricing versus competing vehicles?), Content (features), Eco-Rating (is a gas miser really that good? cough”Aveo”cough), things like that. Broad criteria, that apply to every vehicle, regardless of Genre.
Food for thought.
The question is; Is the Tiguan a 4 star vehicle in general or is it a 4 star CUV?
Do you use the same criteria to judge a CUV, an SUV, a family sedan, a sport sedan, a sportscar, and a luxury car etc.?
Has TTAC clearly defined what it considered to be the “right” attributes to be considered and judged for the different classes of vehicles?
Is the Vantage a 5 star car because it is superior to other high-end sportscars costing over $100,000 or is it a 5 star vehicle because it is a better sportscar than a 350z?
Since TTAC tests fewer cars than CR, and not in 4+ group comparison fashion, I think an absolute grading system would be best. That is, do not rate a car based on the other vehicles in its class. Rate it on absolute position within the universe of all vehicles. This is similar to a hotel or restaurant rating system. While a Toyota Yaris might only score a 2 at the highest you would never expect it to rank amongst significantly more expensive cars.
And by using this method of rating you can actually add value to a cars ranking specifically because the score is directly comparable between vehicle classes. This is all the more relevant as new car buyers struggle to choose between fuel efficient (generally economy-class cars) and what their emotions tell them to buy (sports coupe, V8 sedan, etc).
I like the star system as is. Most reviewers on TTAC do not seem to drive a lot of new vehicles, specially next to each other. I think a five star system is pretty self explanatory. 5 if you really like a car, 4 if its good, has some issues, but its still a car worthy of praise. 3 for a car that does make a good case, even though it has big flaws, 2 for car with a lot issues but still runs 1 for a car thats really irritating and needs to be sent back to the factory. I think every editor is going to have his own opinion, that is why I read the review and try to see if the author explains his/her basis for the star rating he/she gives. Often I do not understand a lot about the reviews, so I just take what I can and stay open minded.
btw. I like to see vivid explanations from TTAC. Jokes are great, I love them, but I also want clarity about the cars handling, Textures, etc etc. That may be pretty difficult though, still I want to get a bigger picture. Anyhow I’m glad for TTAC and I’m always looking forward to reading the reviews.
From a marketing prespective (as in, getting more people to the site), the star system must stay.
From a subjective prespective, the star system must stay.
Occasionally I go back and read older reviews without the star system. Without the stars, it can be a bit more difficult to determine the author’s position on the vehicle. Also, I like reading the single line blurbs that the reviewers write in each category of the star system.
To those of you who disagree with the star system and have not read an older (star free) review, I implore you to dig up an old review – just for comparisons sake.
With regards to:
Because of the inherent drawbacks of CUVs as a class, they should have to be absolutely exceptional to warrant a 4-star review. The entire class only exists because people no longer want to drive station wagons.
While the Tiguan may be superior to a Rav-4 or CRV, how does it make sense with cars like the GTI or Jetta Wagon around? It’s more expensive, handles worse, slower, not much more cargo room, uses much more gas, the interior doesn’t look that great from the pics I’ve seen, etc.
Yes, the star system is subjective … so? Comments about an entire class of vehicles being pointless or less efficient than another class of vehicles should be fair game for reviews, editorials, etc. Whether these realities are detailed by an author during a review (or affect the vehicle rating) should be up to the reviewer.
Also, many customers purchase vehicles that are clearly excessive (wrt to their needs). Therefore, they may completely ignore the more efficient vehicle class and go straight to the CUV class. The Tiguan review would be perfect for these customers. If a TTAC reader/car shopper decides to read the B&B commentary, they will come across multiple facits that are not covered in the review – and will benefit.
As for starflation, nahhh. As long as the reviewer’s views/justifications are clearly expressed, I’m fine with the current system. For example, let’s say that Frank had given the Tiguan a 3 instead of a 4 because of his belief that all CUV’s suck. As long as Frank mentioned his bias/belief somewhere in the review, i’d say it’s good to go.
Any attempt to make TTAC’s star system more scientific would make the reviews/star scale more sterile, less creative, etc. If readers want more scientific reviews, they can head over to Consumer Reports. As [self proclaimed] car enthusiasts, I would think that our values would be slightly different from the mass market. For example (again), if the Tiguan (again) “drives” significantly better than the Rav4, then by the TTAC standard, it would get a nod (with a note on the price difference). At a less subjective site, the “by the numbers” camparison would note the performance (0-60, 1/4 mile) of the V6 Rav4, existence of a third row, and lower pricing. The steering of the Rav4 would be labeled as boring, and with that the Toyota would easily win the comparison.
That was my reaction to the latest “4 star” car review that I saw when I opened up TTAC this morning, but I decided not to bother with commenting. Obviously I changed my mind. Going back one more review in time to the Mazda6 review, the printed word sure seemd to describe a 3 star car. Good in some respects, trailing the competition, if however slightly, in others. How is that a four star car, one that would be considered better than most. Of course, a five star car should be one that is exceptional in most if not all respects, at least in sofar as what it aims to be (i.e. don’t count lack of towing capacity against a sports car or lack of acceleration and handling against a truck).
I think the author’s rating should be the starting point, and we commentators ought to be able to give our own ratings that will be averaged with that.
:)
chinar :
Instead of stars, you should provide the relative standing of the vehicle w.r.t its closest competitors
For example, a VW tiguan review would be summarized as:
1. Rav4
2. Tiguan
3. CRV
4. Vue
or something like that.
That would not allow the reviewer to express his/her opinion on all the vehicles, as they would only be reviewing one on the list. Additionally, the reviewer may not have driven all the vehicles.
And my 5-star review of the Genesis 4.6 was because there is no 375hp, RWD, power everything, sedan at its price point.
Just because it is the only car with these features in its price range is not reason enough to give out 5 stars just because.
The question is; is the Genesis a better car than a G35 or other cars in its price range? Even with its features is it worth the asking price? The Genesis could just as well be a POS that does nothing well but cost less than other cars it has no hope of successfully competiting with. Interesting that the Genesis was outperfromed by a GS350 in an Edmunds test. Edmund’s made the claim the the Genesis v8 did not perform as advertised. The Genesis did win the comparo though.
A rating of 5 stars make the Genesis out to be the as good as the LS400 was when it first came out. IS TTAC making that claim? Is TTAC making the claim the claim that Hyundai has produced a car of that calibur?
On the otherhand if the Genesis can be rated so high what does that mean for the other far more expensive cars that it does compete with in term of category and not price? IS the Genesis a better car than a 535i. I do not remember the TTAC rating on the 535i but if the Genesis is so good to rate a 5 than I guess adjusted for price the BMW should only be a 3 (average)at best.
Sajeev Mehta :
Restricting the writer’s creative input here is a long and slippery slope…that (quite frankly) none of us have the time to deal with.
That’s the motherfucking (opera?) truth!
Good points made by many, both for and against star ratings, as well as those in or not in agreement with the statement that the stars are being handed out too generously.
I have no opinion about it either way, because I read fewer than five reviews each year.
I loved the Mario pic, though. Dovetails nicely with the subject line!
The recent Mazda6 review read like it was a solid little 3-star automobile. I was a bit shocked to see a fourth star at the bottom of the article.
iNeon :
I think the author’s rating should be the starting point, and we commentators ought to be able to give our own ratings that will be averaged with that.
Exactly! People: if you disagree with aspects of a review/editorial, comment on it! Add your subjective/objective views, and we all benefit from lively discussion.
I hate the stars, as the reference point can be quite arbitrary, but RF insists on them to please the readers who like such things. So I give them out.
With the Mazda6 review yesterday, I felt the car was clearly better than average, but not outstanding. So a four. In the end, four probably often means, “Good car, but it’s got a few shortcomings and I’m not crazy about it.” A three would mean a near total lack of high or low points, while a two or one would probably require that we detest something about the car.
Manufacturers are better about eliminating weaknesses than they are about creating strenghts, so you won’t see many ones or twos.
I’d have probably given the Touareg a two if I’d posted a review here, because I actively disliked driving it. But clearly other reviewers here have had a different impression.
Is it so hard to understand that we look upon things differently?
I know that asking people’s opinions on cars should always be taken as advisory…NOT gosple!
Please!
Anybody that takes a review and runs out with a check book is stupid.
Hopefully you have chosen your reviewers with a little quality and trust.
Then, AS ALWAYS, go out and review YOURSELF!
A hard leather seat to one is support to another.
I am doing this right now.
ES 350?
Hyundai Genesis?
Lincoln MKS?
Toyota Avalon?
Acura TL?
I have listened, and still listen, to as much advice as possible, but I have driven each one at least 2…mostly 3 times.
All except the 09 TL, which in not availabler as yet.
But give us a break.
Stars…who cares.
Listen to what is said and go out and award your own stars.
This is supposed to be fun.
Well, lot of good points above.
I’d suggest that there be some kind of criteria for the star rating to make it a bit more informative. It’s a little vague whether the Malibu for 4 stars because its a 4 star vehicle, or because its 4 stars compared to mid sized sedans, or whatever.
And maybe some specific meaning to the stars as well. There have been multiple reviews where the author trashes or praises a car in the text, then hands it a seemingly random star rating.
How about on the homepage, make it so the stars show only AFTER you read the entire review. It kills the curiousity of readers, I think.
And yes, I think there’s definite inflation. There is absolutely no point to the star system unless it provides some gauge of where this car stands relative to it’s competition. It’s useless to give car A 4 stars when car B, C, and D all have 4 stars. Star systems and scales 1-10 exist only to show the relative standing of the cars, I think. I remember when TTAC reviews were much more varied in their conclusions, and far more compelling to read. Compare the Audi S4 Avant review from 2005, for example, to the more recent reviews…
I ignore the stars for the most part, though have noticed a few more 4-star reviews then normal in the past few weeks. I think it’d be pretty difficult to bell-curve (or other) the results since you simply don’t have the all the data until the end of the year (assuming your curve covers vehicles for the year). The only way to successfully do this is to go back over all vehicles each time you rate a new one and adjust all of them accordingly.
I do really like TEXN3’s idea about a relative class comparo snapshot instead of star ratings; I’d see it more like this (example – just making it up!):
Review: 2008 Honda Accord
Better then: 2008 Chevy Malibu due to size, economy, insert witty comments, etc, etc,
Worse then: 2008 Toyota Camry, due to size, economy, insert witty comments, etc, etc…
Or, if you like, give a relative star comparo to cars in it’s class… ie:
2008 Honda Accord: 4 stars, if you consider the Malibu 5 stars, and the Camry 3 stars. This would allow you to “re-adjust” the star ratings with each new review, though it may become a little confusing – certainly unique, though!
I give this thread 3.5 stars.
The recent spate of high rated cars may be happenstance, or signal an editorial policy change to soften TTAC’s hard-ass image. Regardless, I have confidence in the publisher’s integrity.
The Star ratings muddy the water. Is a 3/5-Stars $58,500 Audi S5 really inferior to a 5/5-Stars $40,000 Hyundai Genesis? If they sold for similar amounts I doubt many would choose the latter.
I ignore the stars unless they are 1 or 5. In either of those cases I know I am about to read a fun review.
The testers’ impressions, conveyed in readable prose, are what counts.
So if Tiguan, Rogue, Challenger, CSX, Accord, Mazda6, all get 4 stars, how many stars for the Maybach Landaulet?
Gardiner Westbound :
September 10th, 2008 at 3:54 pm
The recent spate of high rated cars may be happenstance, or signal an editorial policy change to soften TTAC’s hard-ass image. Regardless, I have confidence in the publisher’s integrity.
The Star ratings muddy the water. Is a 3/5-Stars $58,500 Audi S5 really inferior to a 5/5-Stars $40,000 Hyundai Genesis? If they sold for similar amounts I doubt many would choose the latter.
Yeah, but the Genesis is probably a better 40k car than the Audi is a 58.5k car. That is, the star system should factor in the vehicle’s place in the market place.
And it’s not just cost. If you are reviewing a minivan, how “kid friendly” it is is more important than how it handles. But if you are reviewing a sports car, the opposite is true.
The real problem with the star system is this: It doesn’t jibe with the individual categories. This makes the overall rating useless. In theory (or so I thought), the overall rating should be all of the individual categories averaged out, but in reality this doesn’t happen. If the overall rating doesn’t agree at all with the individual ratings, what’s the point of either one?
whatdoiknow1 : Just because it is the only car with these features in its price range is not reason enough to give out 5 stars just because.
Those are not the only reasons, I just posted a few to make the point.
The question is; is the Genesis a better car than a G35 or other cars in its price range?
Sure it is, if you prefer a smooth ride, effortless V8, a leather wrapped dash, etc. Some people actually like that stuff and I happen to be one of them–on certain occasions–even if its not a driver’s car like the G35.
The Genesis could just as well be a POS that does nothing well but cost less than other cars it has no hope of successfully competiting with.
It doesn’t drive like the bleak picture you paint for it. It drives like a 4-5 star car. And maybe I gave it 5 stars just to round up, because 4 stars isn’t enough for a car this good.
Interesting that the Genesis was outperfromed by a GS350 in an Edmunds test.
Sure it did, revving and torque steering all the way. I’ve driven both, and the ES350 is not a luxury car just because of the Camry bones and massive torque steer which requires constant attention in a straight line, much less a corner.
My criticisms of the Genesis’ throttle mapping is in print: if they fixed it the Hyundai would absolutely cream the Lexus.
A rating of 5 stars make the Genesis out to be the as good as the LS400 was when it first came out. IS TTAC making that claim?
How did you interpolate that from 5 little stars? I mentioned The LS400 in the review, but that’s not the measuring stick for the Genesis’ attributes.
I do not remember the TTAC rating on the 535i but if the Genesis is so good to rate a 5 than I guess adjusted for price the BMW should only be a 3 (average)at best.
I don’t get your logic. The BMW is smaller, more expensive, has less peak power, rides firmer, etc. Just because the Genesis is a cheaper land yacht from a low-brow automaker doesn’t mean its not a 5-star car for people who like luxobarges.
Not to mention the Genesis doesn’t have the legacy of the E39 shaming it. The new 5’s style and i-Drive are big turn-offs for me.
I vote yes.
The Tiguan review would lead me to believe it would get 2 or 3 stars at most, but then it pops a 4.
I won’t retread all the arguments made by better writers than I, but I think it needs a re-boot.
It also may be that often the reviewer gives too much space to the negatives, and so the Star Rating seems incongruous.
Why not let the writer post his review, and let the readers vote for the Star Rating. I think then you’ll have more input on how they are perceived.
whatdoiknow1:Interesting that the Genesis was outperfromed by a GS350 in an Edmunds test.
Sajeev: Sure it did, revving and torque steering all the way. I’ve driven both, and the ES350 is not a luxury car just because of the Camry bones and massive torque steer which requires constant attention in a straight line, much less a corner.
The Edmunds test was against the GS350. The people at Edmunds considered it the RWD luxury car closest in price to the Genesis V8.
The whole problem with the percentage quota as in the main question is this: Cars MUST be rated by a category.
You can never award a great low end car IF it has to compete against the 200K cars.
Sure, there is nothing like a 200K car, BUT NOBODY CAN DRIVE IT!
You must be able to award a 5 star rating to a great small, economical, reliable, fun masterpiece.
Whatever your qualities…
If somebody was able to build the best small econimical car, hitting all the stated goals, then YES, award them for the task well done.
1 star…hate the damn car
2 stars..pretty crappy car
3 stars..an OK car
4 stars..pretty good car
5 stars..really great damn car
So come on here, what’s the problem interpreting somebody’s stars?
And the Genesis deserves the 5 stars…TO Sajeev Mehta.
Me, I would have given it 4.
Or 4.5.
But so goddamned what!
DON’T run out and buy one just because!
Drive it and award your stars.
Its your job IF its gonna be your car.
I think there has been some grade inflation. We had this discussion regarding the 3 star cars a month or two back. People need to give out 1/2 stars in their ratings, and some turds deserve a zero.
A “Better than, Worse than…” ranking would help, as would a few bullets unders “Highs” and “Lows.”
Comparing cars in categories and ranking them would also be nice.
If the author can write a good 800 word review, they can handle rating the car on a 1-5 scale. come on, give these guys and gals a little credit
I want to see the star system removed. It doesn’t really mean anything, it’s just something for people to argue about. All critiquing of a car should be within the review itself.
All reviews are subjective. I’d rather see the continuation of very frequent vehicle reviews rather than fewer reviews necessitated by our fine reviewers getting bogged down with detailed evaluation reports and such.
But I must say, if the 2008 Dacia Logan MCV 1.5 dCi gets a 3 star rating, then you just about have to give most other players a 4 or 5!
Big smile.
I have no problem with the present star rating–I know there is subjectivity. I just enjoy reading the reviews since I can’t drive everything and can’t aspire to much more than my practical Civic. My main enjoyment on this site comes from the news, editorials and ensuing comments anyway!
I guess as a reviewer you need to determine what a movie/car is trying to be and then judge how successful it is in that endeavor.
What if a car was built to be a complete turd, and succeeded mightily at being one? :-)
Like the old saying about trying to fail and succeeding?
If you want to have any credibility there better be something that gets a 1 star rating.
Have any of the TWAT been reviewed? Did any TWAT get anything other than a 1 star rating?
(edit that however you like I’ll still think of the ten worst cars as that.)
Take a look at page 2 of http://www.jonnyguru.com/modules.php?name=NDReviews&file=print&reid=65 and look at the tables with the word FAIL in them. Page 3 where they show the burnt parts. Page 4 where they give it low scores on the scale of 10.
I don’t want to see TTAC doing the stupid 15 page review that tech sites do to increase the number of page impressions per article but I do want to see you take someone to task for whatever bonehead product they put on the market.
I’m not suggesting that tech site or any tech site should be your model for reviews I was just looking for a quick example of a review where a very low score was given.
You don’t have to grade on a curve. Just don’t cherry pick the cars you review. Pick the worst car on the market and make an example out of it at least once a year.
If you want your reviews to have balance pick something still on the market from the https://www.thetruthaboutcars.com/the-truth-about-cars-ten-worst-automobiles-2007/ and review it. If these cars have reviews on TTAC then hotlink to the review from the ten worst list.
The Aveo got 2 stars but it’s on the ten worst list. Isn’t ten worst list status enough to merit deducting a half star or something? https://www.thetruthaboutcars.com/chevrolet-aveo-review/
Unfortunately, I have only a short lunch break, and do not have time to read all 65 posts above, so apollogies in advance if any of my tuppence worth is repetitious.
The purpose of the stars is, presumably, to propvide some sort of quantitative measure of a car’s ‘goodness’. Well, as I think has already been said, TTAC works by being humourously, succinctly critical, and the meat of the review is always in the text. AS we know text does not well distill down to a five point scale.
My suggestion, in striving to provide comparability, would be to break the review down into topics, a la http://www.parkers.co.uk, where driveability, equipment, safety and stats etc are all separate paragraphs with headings to alert the reader (but not separarte pages).
Seeking to provide a glib 5 point rating to compare all aspects of all cars is doomed to fail, least of all because cars have varying missions. The car must be rated on its achievement against its mission.
Personally, I don’t even read the stars for this reason. I read the text and decide for myself. That is the beuty of the written word.
Further, and it’s not lunchtime anymore dammit, but no car is discretely 3/5 stars. My Pug307 is easy to drive and the ride’s good with decent economy, but I find the engine somewhat tepid, and the steerings too light. I still think the car’s fit for purpose, which is tooling round town and my 4mile commute to work on wet days when I don’t want to cycle. Elements of any car may be 1 or 5 stars, communicating this by icons is never going to trump a 150 word paragraph on the driving satisfaction, and nor can it take into account that, for a Porsche, it’s below par.
Sorry for typos, no time to proofread. thanks fo rthe Forum, RF.
nichjs
Portsmouth, England
The star rating is fine as is, as long as the review is as informative as possible.
Star ratings on the basis of 5% best cars etc is impossible IMHO.
An example that comes to mind is the Bullit Mustang. ‘Objectively’, it has a crappy interior, an ancient live rear-axle and an average engine (I think 99% of the people here would agree, so that’s why I say objectively). In fact, if average is 3 stars, just looking at the above qualities you might give it 2 stars.
However, what it should do, for the market it is intended for, it performs really well, regardless of it’s shortcomings. So well in fact I think 4 stars are perfectly validated (I don’t know what it got on TTAC).
Still, is it a better car than the average car? For most people, obviously not.
I also agree that the review itself is a lot more important than the star rating btw…
The last six reviews ALL got 4 stars.
The AVEO got 2 stars, for cryinoutloud!
Is there star inflation?
You have to ASK???
Many have hit on the solution…
CONTEXT.
A Mini Cooper S can have 5 stars.
A Porsche Carrera GT can have 5 stars.
Whoa. Hey, how can they both have five stars? One is a supercar at an unobtanium price. The other is a small economy car.
But one can argue that when compared to cars within their own classes, they are among the best.
Likewise, a Mazda Speed3 can have five stars. And a Mercedes R-class can have two.
Is the Mazda a better vehicle overall? Probably not in terms of amenities, fit and finish. But it represents its class much better than the R represents its own.
Domestic Hearse
EXACTLY!
Why is this so hard to get?
Cars need to be judged by their position in their own class and the success of their stated golas.
Otherwise you end up comparing cars against each other that had totally different goals, totally different targeted consumers.
If the goal was for Ferrari to produce an affordable, gas efficient off road car, they failed!
I agree with Phil Ressler — get rid of the star ratings. TTAC doesn’t have the editorial infrastructure to create a ratings scheme that is “empirical” enough to mean something (at least to those with statistical background). I wonder whether you’d want to go that route anyway.
Seems to me what you “sell” is context grounded in the subjective but thoughtful opinions of individual reviewers. The star system works against that branding because it implies a comparability across reviews.
TTAC reminds me of C&D of yore — its greatest asset was the personality of its writers. Keep that in the forefront. Star ratings muddy the waters.
Thanks for asking. Since there seems to be near universal contempt for grading on a curve, why not grade the way grading was supposed to happen? 0-100%.
Interestingly, I’ve seen video game reviews done this way, and if something cracks 90%, it really is a terrific game.
90-100 – Truly outstanding, class leading or creating a new genre. Aston Martin, Mini, Porsche Cayman, etc.
80-90 – Very good to excellent. Near the top of the class, but slight improvements needed. Audi A6, Corvette, Genesis, Accord
70-80 – Good to very good – first gen cars that need considerable work to be considered class worthy, but a good first try, or cars with history but haven’t reached the top of the segment. VW Golf, Passat, G8, etc.
60-70 – Below average to good – Chevy Aveo, Cobalt, etc.
50-60 – Why did the mfg bother to below average – Oh, let’s see how many Chrysler products fit here. Hmm. How bout all of em.
Less than 50 – You shouldn’t be reviewing unless as a joke – Yugo, Hyundai Excel from ’89, Suzuki Samari, etc.
I’ll be honest – I don’t even look at your star system – half the time its inconsistent with the review itself.
You guys should make a huge list of cars, and then insert cars where they belong in order. Also make the list dividable by category. As new cars come out, then they can get rated appropriately. The problem with star ratings is a car that is worth 5 stars, can easily be superseded by another car that also gets 5 stars.
Or you can just give them all numeric scores behind the scenes and then normalize them to stars on a bell-curve. This means scores will shift as new cars are added, but it also means you get rankings that reflect the state of the market.