Does motor oil break down over time with little mileage or engine hours?

No, oil does not break down just by sitting in your crankcase. But you have been given a number of reasons favoring an annual oil change nevertheless. A change may not be totally necessary at this time but it is a sensible precaution.