Vacuum leaks that come and go like that are unusual and would usually result from a faulty device like a problematic brake booster or a power steering pump vacuum valve. You’d notice the problem tracks with pressing on the brake pedal or turning the steering wheel. Or something else that is solenoid activated and supposed to affect idle speed, like turning on the AC.
Edit: Unless this engine has an idle speed adjustment screw which has been adjusted to correct a problem that shouldn’t be corrected with the idle adjustment screw. Doing so can cause the idle speed to be unstable.
Since you aren’t noticing any correspondence like that, the EGR advice above is a good place to start. When I’m testing the EGR on my Corolla, the first thing I do is make sure when I activate the EGR with a hand held vacuum pump it stalls the engine.
There’s a chance this is spark or fuel related too. A mechanic could rig up a fuel pressure gauge on the fuel rail and watch it as the car was driven. If the pressure drops at the same time the idle speed drops, that would indicate a fuel pressure problem. Doing the above is best left to a pro.
If you want to do something yourself that doesn’t take much time, you could inspect the vacuum line which goes to the fuel pressure regulator. Are there any signs of gasoline making it into that line? If so, the fuel pressure regulator likely needs replacement.
It would also be wise to have all the ECM diagnostic codes read.