I know that the starter motor temporarily turns the flywheel to help start the engine until it can run on its own but when does the starter motor know to stop turning? What mechanism stops the starter motor once the engine is running by itself?
When you turn the key to “start,” the starter kicks out the gear, which engages the flywheel and turns the engine. Once the engine starts, the driver releases the key, which disengages the starter from the flywheel. The starter is not constantly engaged to the flywheel. Basically, the driver stops the starter once the engine is running by itself.
On Bendix? starter drives, which hardly anyone uses anymore, the gear was on a spiral thread and the inertia of the starter’s acceleration would screw the gear out to engage the flywheel and the engine starting would screw it back away from the flywheel.
In addition to mark9207’s comment, if the driver leaves the starter engaged any longer than needed, the engine’s flywheel can spin faster than the starter gear. That’s not a problem because a one-way clutch in the starter gear prevents the starter from “being driven”.
Without this one-way clutch, there would be lots more starters with windings ripped open from centrifugal force.