Brake noise in the rain only

I don’t how to explain it but this seems different. I had a Neon and the rear brakes made the scrape the rust sound every morning. This doesn’t happen until the brakes get wet. And the car seems to pull to the left when it happens. After it starts, it does it 1, 2 or 3 times then everything is fine. I live on a gravel and dirt road and I wonder if that has something to do with it.