I have had good results with CD2 detergent additive, not the thick honey version, for quieting a noisy lifter. Worth a try before you spend bigger bucks.
A little lifter noise will not harm your camshaft but a lot can do that.
If the lifter quiets down after the motor warms, I’d ignore it. It might not be a lifter; could be a worn rocker arm pivot assuming that you have an OHV engine.