I had a similar problem with my ES300, same symptomes, turned out the problem was with the starter solenoid, easy and cheap to fix. There are two contacts and a disk/rod assembley in the solenoid that wear out over the years. Replacement contacts were $15 at the dealer, fixed the problem for 2 years, then had to replace both the contacts and the disk/rod. You can google “toyota starter rebuild”.
Here’s a link showing the parts: http://www.team-integra.net/forum/display_topic_threads.asp?ForumID=15&TopicID=123037