“Out of several million readers, that is still a lot”.
The number in the sample is not the critical issue. The randomness of the sample is the important criteria. Here are two examples:
In 1936, a publication did a telephone survey asking the respondents who would be their choice for President of the United States in the next election. This was a nationwide telephone survey. The results were that a huge majority of the sample favored Alf Landon for President over Franklin D. Roosevelt. Well, even though U.S. history textbooks are suspect, I don’t think there was a President Landon. I do barely remember my parents saying that the President had died and it was FDR. In 1936, the nation was in the last part of the depression and the less wealthy did not have telephones. Rural areas were also not served by telephone companies. Had only telephone subscribers been allowed to vote, there would have been a President Landon.
In 1955, Ford Motor Company did an extensive survey as to what price range and features automobile buyers wanted in their next car. The result was that in late 1957 the Edsel was introduced. It should have been a smash. However, the country had gone into a recession and autombile purchasing habits changed. Either people held onto their present cars or they opted for less expensive cars. American Motors became profitable selling Ramblers. Even Studebaker sold twice the number of the stripped Scotsman models than it expected to sell. Had For brought out the Edsel immediately after the survey was taken in 1955, we would probably be reading about problems of 2008 Edsels on this post today.
Consumer Reports is a good publication. The frequency of repair ratings is a good place to begin when considering a car. However, one needs to realize that 1) people who don’t subscribe to Consumer Reports may have a different opinion than those who do and 2) the higher percent of Consumer Reports subscribers who do not return the questionnaire may have a different opinion than those who do.
Consumer Reports now reports owner satisfaction rating as well as a reliability rating. When I get a chance, I am going to correlate these values and see if there is a relationship between owner satisfaction and reliabiliy.
Good post!! The polsters made the same telephone survey mistake again in 1948 when they predicted and went to press that Dewey won the election; many of the democrats who voted for Harry Truman (who won)did not have phones!!
An accurate sample of the whole US popuplation of over 300 million needs only about 15,000 interviews. That’s only a tiny fraction of a percent!! But, as stated, the sample must be choses to represent the whole country proportionately.
So, if CR has their samples correct, the reliability reports are correct 95% of the time. If you go through back issues of CR reports, and talk to your mechanic you will find that the troublespots are very accurate.
When I was in Business School, our class did a travel survey of single women and their choices of destination and mode of travel. A major airline sponsored this project. Our school was located in a city that was representative of the whole country, sort of Cleveland, Ohio in the past. I had the arduous task of interviewing 30 unattached college females; somebody had to do it! In total, about 400 individulas were interviewed, and the airline based their successful promotional campaign on our findings.
go with what ya know.
you say you’ve had a great relationship with your current bmw 3 series. i’ve had a couple myself. (i change cars more often)and found them to be very reliable, hold their value well, good looking and most of all lots of fun to drive.
so what can i tell you buy another 3 series, in fact treat yourself to the convertible. you can get it w/ rollover bars, the top is well insulated against temp and noise and the fully automatic top is 1 button user friendly. go for it!!!
Even if they have their sample correct (good distribution of owners), though, sample sizes of 200-300 (per model) are not large enough to determine differences between two populations when failure rates are small and where the measured difference is 1 or 2%.
That’s the problem. You have inaccuracies even in a perfect sample that make it so that they need larger samples. Then you have inaccuracies because their samples are not properly chosen (too low of a response rate).
That doesn’t mean that the data is useless, though. You can easily use the very same data to show that failure rates are, indeed, small.
So a vehicle with good ratings is almost guaranteed to be very good on average.
But the relative “average” which hardly differs from “excellent” is so close to “excellent” that you can’t measure the difference with accuracy…
Good point shorty! On their appliance reliability & failure data, CR states that “differences smaller than 4 points (% trouble) are not maningful”, for instance on the refrigerator survey in May. That was a sample of 74,000 fridge owners, and the top 4 brands all fell wthin that range. Only Maytag stood out as a dog in most models.
I have always had faith in the figures for high volume cars, since the samples would be much larger, and troublesome cars like Mexican-built Volkswagens consistently score badly (worse or much worse than average). I agree that a low volume car would have results that statisticaaly would not be reliable.
I also subscribe to TrueDelta, and the “in the shop” figures correspond well with CR data.
n the article I read, PM had surveyed owners of the Ford and corresponing Mercury vehicle. In the demographics of the owners, the Mercury owners were 7 years older than the Ford owners for that particular pair of twins. My guess is that the same the was true for the Ford Maverick/Mercury Comet pair. The younger owners bought the Maverick and probably drove them harder and didn’t have the money to spend on maintenance. This age factor, IMHO, contaminates the results.
If you are looking for a new car, then yes this could be considered “contaminated” results, but if you are buying used those are good results. It says the Ford buyers don’t maintain as well, on average, as the Mercury buyers. Good to know when looking at used cars.
We use better methods than the others, but our sample sizes are in most cases still small. So I wouldn’t try to split hairs with most of these results, just get a general sense of low, medium, and high–and of how often cars tend to require repairs. You’ll find that in most cases we’re talking about less than one repair trip per year.
The big advantages of our results:
–actual repair rates, not just dots
–continuous tracking, not just a couple of points in time
–prompt updates four times a year
A dose of reality…
We talk about reliability as a selling point…in reality it is the last thing that auto manufacturers want, or consumers for that matter. Selling cars that are too reliable and you decrease the turnover rate. Engineering gamemenship is the name or the game. The ultimate motor (electric) will never see the light of day from automakers by choice w/o IC back up. With upwards of 40% of their profit in maintenance, only John Deere, Kabota and the like are qualified as truley reliable vehicles. The “family car” is a joke where reliability is concerned.
Build a car like a hybrid locomotive and you eliminate transmission/braking problems with power to spare.
Look at the “family” farm equipment that is handed down from generation to generation to take your que as to what American manufacturing is truely capable of… Henry Ford would “give” cars away just to own the parts market.
Toyota/Hondas are only as reliable as they have to be as it remains a selling point to steal GM customers. They all play the same game and cars are no different than other “replaceable” consumer goods.
The biggest difference is not the make of the car but rather the owner and they care they give the car. Any car is a matter of luck some will do very well and some, even the next one off the line, may have a number of problems, but that difference is nothing compared to the owners and their care of lack of it when it comes to dependability.
I agree that the best, but still flawed, data on cars is Consumer Reports. It is flawed because it is based on owner responses. It would tend to follow that certain cars tend to attract certain kinds of owners. For example performance cars will likely get owners who may push their cars more and may well see more repairs than the exact same car that was owned by a more conservative driver.
And the higher end cars will have pickier buyers as well. If I paid 60 grand for a car, you better believe that I’ll nit pick every little squeak, rattle and shimmy.