Did you csll the credit card company? What did they say?
You messed up by letting your card be debited for this in the first place.
Challenge the whole amount, rental included. Tell the card co. that you think the car was lent under false pretenses (i.e. so that they could ding you for prior damage) and you have no desire to honor a contract written in bad faith.
What can they do to you? I don’t think they can even hurt your credit rating until the dispute has been resolved. No way are they “going after you” legally.
I suspect that if you do this, the rental co. will quickly cave into a “at least let us have the original rental fee!” position. Then you get to decide if that’s acceptable or not.