Sun Visor

Ok, this is annoying. The driver’s visor keeps working itself off the rod. Almost every day, I have to whack it back by hitting the tip of it with my fist. What needs to be replaced? It’s not sufficiently annoying to go to a mechanic if I can do it myself. Thoughts? Thanks very much! Deb