Actually, I don’t think that the OP is using manual mode.
On the newer Outbacks, in addition to using manual mode, one can downshift with the paddles. The difference is that when doing this while in Drive, the downshift is temporary, and the trans will automatically upshift shortly.