@texases The samples are statistically meaningful, however the type of owner and driving style will influence the results. If the sample is too small CR will put an asterisk in the column. That measn “insufficient data”. Most of the CR staff have passed “Statistics 101”.
Some years ago in the period when all GM full size cars were cookie cutter same models with only trim differences the results were very different in the CR survey.
The most reliable ones were Buicks, followed by Oldsmobiles, then Chevies and Pontiacs were the worst by far! How Come??? Well, it seems to have a lot to do with type of BUYER. Buicks were promoted as quiet classy cars and were bought by that type of older buyer who did all the mainteance and drove conservatively. Olds had a bit faster image, but still a comfortable middle clas car; TV star Dick Van Patten, a real casper Milquetoast type, was it spokesman.
Chevies were the workhorse and got a good workout with average maintenance and were not babied. They had an average number of failures.
Pontiac in those years promoted ATTITUDE!!! Their ads showed it and the type of buyer was a guy who really wanted a Firebird, but had to settle for a full size sedan. This type drove with attitude and had a more casual attitude toward maintenance. My son worked with such a guy and he had no end of trouble with his Pontiac.
So, even statistically correct samples can produce different results base on different drivers. That’s no reason to throw out the results. CR will recommend any car that meets all their driving and safety factors and has at least an AVERAGE repair record. That’s why they will not recommend any brand new model; the last time they did that was with the new 1986 Ford Taurs, a nice car but full of mechanical flaws.
By comparison, the poplularity of a US politician is determined by surveys with samples of as little as 2000 respondents, but the samples are carefully selected from clusters across the country to represent various economic, racial and ethnic groups. Gallup says such results are 95% accurate 9 out of 10 times.
Any difference in driving style should be washed out in the stats. But there are MANY vehicles I’ve seen that where driving style had NOTHING to do with their differences. When you look at the numbers as a whole…they seem fine. But some of their individual cars their data samples are too small and can have errors.
I am a longtime CR reader, and recognize that there are some flaws in their methodology, without being able to come up with a better way. Two have been mentioned. They do sometimes report on inadequate numbers of replies. That does sometimes cause odd results for essentially identical midels. Which might then swap places the next year. Then there are ones I never solved. For quite a few years (a few years ago) they consistently rated the Nissan Versa hatchback as above average and dumped on the sedan as a turd. They were made on tge same assembly lines and differed only in the trunk area. That was a total mystery, but it was consistent for several years. All I coukd figure was that the hatchback cost slightly more and may have had more older, cautious drivers.
But my main problem with CR is that their members are always right. If they say Car A is better than identical Car B, it is reported as a real difference and they rarely dig deeper They should look harder at the demographics and at least report that info to the reader. Toyota buyers now average a decade older than Honda buyers. That matters. It wouldn’t be hard to report age, sex, and income category for each car.
They do have a marking they added a few years ago to ratings that are based on fewer responses than typical, but it took a lot of nagging by readers to get added. That’s distinct from the asterisks for insufficient responses. I also think they don’t make enough adjustment for type of vehicles. Luxury and performance cars have more parts to fail. Mostly minor comfort and convenience features, but those malfunctions have too much weight in their rankings. Dead simple vehicles, sometimes so simple because they’re dated, do better in their ratings. Because of that you can only use their ratings to compare vehicles that are the same size and have similar buyers. Which is OK, since most people are doing just that, but for someone open to a range of possibilities the ratings have to be used with caution. You can compare a Focus to a Cruze, but not to a RAV4 or BRZ.
@MarkM Good thoughts! I spent a number of years doing market research, but never let the survey results alone decide on a course of action. When Ford did a focus group survey with the first Mustang, they found the couples interviewed were lukewarm to that car, since they thought it would cost around $7000 or so. When Ford told them it would be around $3000, suddenly nearly everyone was interested. Lee Iacocca already knew there were a lot of portential buyers for a sexy car that was as easy to take care of as a US compact.
As a result, the Mustang became an all time first year best seller, and was probalby the most overrated car of its time.
Your point that large complex cars have more problems in understandable, but that general fact is very useful for potential buyers. It tells you in advance that you should expect more expenses. Again, however Lexus seems to defy these observations, since nearly all there models rate well, regardless of complexity.
The value in the CR results are in the longer term which show that certain models are more prone to problems than others. The type of use the car is put to and the driver also contribute.
I use at least 3 sources of information to get a handle on the reliability of any car I’m going to buy.