There is nothing inherently unsafe about driving a car without power steering. In the early days that was the only kind the made. Of course, steering ratios were different.
You are already aware that when driving at speed, the effort to turn the wheel is about the same as with normal power assist. More effort is required at low speeds, when safety is unlikely to be much of a factor. As long as you confident you can deal with the steering you may continue to drive.
If you can determine that the steering pulley spins freely you need not worry about the belt.