The symptoms described don’t sound like an O2 sensor problem. I’ve never heard of anyone cleaning one, but I suppose a person could try it. If the cleaning treatment ruins it and it starts throwing a code, inexpensive to replace with a new one. I don’t see much down side to it. But I don’t see much upside potential either. For reduced mpg the OP would be more efficient spending diy’er time to make sure the coolant temp is getting to the manufacturer’s specifications for a warm engine. Thermostats eventually warp and don’t close all the way, which can reduce the coolant temp slightly. And even a slight drop in coolant temp will reduce mpg.
If OP decides to clean the O2 sensor anyway, probably both the outside (the part outside the exhaust stream) and inside should be cleaned. O2 sensors have a passageway to allow outside air into the sensor part inside, and that can get clogged up and prevent accurate O2 readings.