Yes, I think driving with bad rear brakes is dangerous and a bad idea. It sounds like you’re metal to metal on the rears.
Think of it this way. 100% of the inertia of the vehicle is stopped by the brakes. 100% of it is converted to heat energy, almost all of that by teh brakes. At 70 mph that’s a lotta inertia.
In a proper functioning system, it’s done in a balanced way, with about 70-80% of the energy being dissipated by the front brakes and sufficient load on the rear tires to keep the rears under control and the car going straight. If you have no pads on the rears, 100% of the energy has to be borne by the fronts, and there’s insufficient friction on the rear brakes to be allowing the rear tires to be grabbing the pavement to ensure proper control. Your front brakes and your front tires will be doing more than their share of the work.
You may get away with it. But if you have to make a hard braking evasive maneuver you’re screwed. It could cost you your life.