The reason the car compass works sometimes and sometimes gives a false direction depends on which way you are going and magnetic lines of force conducted by the roof structure and firewall (and often the dashboard supports).
These parts are made of steel. Steel becomes magnetized in the presence of a magnetic field. So, depending on the direction these parts face, they can carry magnetic fields to the compass from the wrong direction. This causes the compass to point in the wrong direction.
That sounds about right; but, also, some people just have magnetic personalities. They could be distracting.