i have a 2005 Subaru Outback 2.5 XT with this EXACT same issue. Has been happening on and off for over a year now. I was not able to replicate it at a garage until just yesterday. I would describe it as a reduction in power or long hesitation while attempting to accelerate, most noticeable within the first few minutes from first starting the vehicle. I never get on the throttle enough to engage the turbo so I assumed the turbo was not the issue (although there seem to be PLENTY of other issues with Subaru Turbos........).
Finally yesterday, I had it into my mechanic. He said Cylinder #2 is mis-firing. Went on to replace the coil and spark plug. This did not fix the issue. This morning he did a leak down test and found that 2 exhaust valves are "burned out". I'm not very car tech-y (much better with bicycles and things that are man-powered), but it sounds like 2 of exhaust valves related to Cyclinder #2 are burned out and need replaced. In addition, ALL OTHER valves need an "adjustment" (which should have been done as part of routine maintenance at 105,000 when the previous owner had the car, but did not happen......).
We're talking ~$3,000 to get 2 burned out valves replaced, 14 remaining valves "adjusted" which means head gaskets, shims, valve "seats", and the heads have to be machined. The labor will be half or more of the total bill.
As of this moment I am exploring other options -- trade, new engine, etc.. The vehicle has 150,000 miles and has been perfect up until this point.
Anyone have any suggestions? Does this sound reasonable? I consulted with 4 other garages, one of which specializes in Subaru Repairs, and he says he sees this "all the time" as a result of skipping the recommended valve adjustment at 105,000 miles.
I'm worried that spending $3k now will only allow the engine to last long enough for the Turbo to Blow, which it sounds like is a good possibility within the next 15 to 150,000 miles......