It's not contrary to what I expect or what I know from experience.
While car makers don't publicize it, when one opts for leather upholstery, the seat structure is almost always more...robust...more comfortable...and more durable. Similarly, while they don't publicize this reality, opting for a more "luxe" model usually yields more sound-deadening insulation, which makes for a much better driving/riding experience.
In my younger years, I scrimped and saved as much as I could, and the result is that now, when I order a vehicle, I can opt for almost every upgrade/option, and that expenditure definitely yields a lot of pleasure for me.
My 2011 Outback 3.6R Limited has really good acceleration, very comfortable heated leather seats, and an excellent audio system that is integrated with its Bluetooth capability, its onboard GPS, and its backup camera. The result is a seamless integration of those capabilities, which I can access (or not!) while riding in a very quiet, fairly fast, good handling, and very comfortable vehicle.
If I had just chosen the base model, NONE of those advantages would be present. Even the brakes were upgraded in this model.