I am amazed at how dirty the cabin filters become in my vehicles. These cabin filters get as dirty as the furnace filters in my house. I would advise keeping the cabin filter in the car and replacing it on a periodic basis for the same reason that a person periodically replaces the filter on a forced air furnace.
Just a side note here–the church I attend has air handlers in a crawl space and the filters were in the return air pipes in the crawl space next to the air handlers. Another person and I got the idea to install registers in the air returns so that we could easily replace the filters without going into the crawl space. Two filters would be dirty when we would replace them, but one filter remained clean. We discovered that the return line for this air handler had a bad leak in a remote part of the crawl space. The air filter that was never dirty tipped us off to this problem. You want air filters, whether in your home or car, to get dirty. That shows the system is doing its job to provide cleaner air.