This should be a rather simple fix. But first do these tests. Slowly move the shifter from N to D4 and see if the D4 light even blinks. If it does blink, then try to find a spot between the selections where the light will stay on, this is two different things here.
If you can find a spot where the light stays on, then you have a linkage issue. If it just blinks but there is no sweet spot to keep the light on, that is the worse case, hope you don’t see that. If it doesn’t come on at all, then there is a switch pack next to the shifter under the console. Chances are that the D4 switch is defective or the wire going to it is broken. This would be another cheap fix.
The D4 light is different from all the others. If all the other lights come on when their corresponding position is selected, that probably rules out the linkage. The wiring for all the other positions is direct, power to switch, then to light bulb in dash. The D4 goes to the computer though. The D4 serves as the check transmission light, but it works by blinking, no going out completely.
If the D4 blinks, then you need to get the code read by the dealer. It is done by shorting out a connector under the dash which will cause the D4 to flash in a series of long and short flashes to give the code. Since your D4 is not blinking, that means the PCM has not detected any faults in the transmission. So that is a good sign, how ever you might have to verify that the D4 bulb is not burned out.
If it turns out that the D4 bulb is burned out, you will need to replace it so you can do further troubleshooting.
Some more test to do if you haven’t done so already. At 60 mph, switch between D3 and D4 and se if the rpm’s change on the tach. At 60 mph in D4, the tach should read about 2450 rpm. Vary the pressure on the gas pedal very slightly up and down to see if the rpm’s move independent of the speed. If it does vary by 300 rpm or more, the torque converter is not locking up. If you are over 3000 rpm, then you are probably stuck in third gear and the transmission is not shift to fourth.
One last test, start the car and watch the D4 light. It and the check engine light should both come on for 2 seconds, then go out. If that happens, then the problem is most likely in the switch on the shifter. It means that the computer has checked the circuit. If it doesn’t come on, then open the hood and remove the cover for the underhood relay panel and pull the fuse labeled radio/backup or just radio. After 10 seconds, reinsert the fuse and start the engine again and see if the light comes on for the required 2 seconds.
If the light does not come on, you have a bad bulb or one of two grounds from the PCM are missing. You will probably need to go to a dealer to get the pin outs and procedures for checking those grounds.
You probably do not need a new transmission so be wary of that diagnosis.