Holes are stress concentrators - and since the sidewall undegoes a very complex motion - so complex patches tend not to hold - the hole become the place where the crack will start. What you want to avoid is the sudden sidewall failure at high speed. Cracks will take a while to develop, but they eventually will fail the tire. So it’s best to replace the tire.
If you want to know more about stress concentrators or crack propagation, Wikipedia does an OK job of explaining them.