My 2006 Sienna has two of those formerly-known-as-where-the-lighter-was accessory chargers in the dash under the gear shifter. Both have come out of their mountings. They work otherwise, but they are not stuck in the dash anymore, if you know what I mean. Can I just glue them back in with ? Superglue? Gorilla glue?
Super glue should be fine. The gel type might be a better choice than the liquid type. Just be sure not to get any on your skin and accidentally glue your fingers together. 
I prefer Loctite waterproof adhesive for jobs like this.
Typically there is a barrel behind that screws on to keep them in place. If you can reach them and tighten them that would be my optimal solution. Chances are the wiring is now twisted so disconnecting the wiring first is advised.