Fuel - shall I be concerned if I don't drive much?

I am worried more about driving more with $4 + gasoline.