Maybe check to see whether that JVC radio has a setup mode that allows you to set it up either way: turn power off with key or leave on.
With the battery having been pulled out, it likely went to some default that doesn’t turn it off.
Also, are you sure it doesn’t turn off? Some have a feature where it shuts off when the door opens or it shuts off after N number of minutes with the key out.