Does motor oil break down over time with little mileage or engine hours?

I drive my car about 4000 miles a year. Driven 2-3 times a week
The Operator’s Manual says change the oil every 6 months or 5000 miles.
So I change it every 6 months.
However, I change the oil filter once a year.
I don’t think water and unburned fuel get trapped in the filter.