Oh, you bring back memories! I applaud you for not hitting him, I could list all the horrible effects it has on him (including lowering his I.Q) but there isn't enough space:)
My son was exactly the same. I honestly though I would have to give up ever having a plesent day with him, but you'll be happy to know...he's now 5 and we've had a very wonderful child since he turned 3 and we changed our parenting.
First...lay off the sugar!!!! VERY important, just try for a couple of weeks...watch everything, juice, popsicles, cookies, jam, etc. This alone made a huge difference with my son (and made him healthier) and now he can handle 'treats' with moderation.
I also found that most of his outburst came when he was either bored or wanted his own way. Curb the boredom by challenging him (not with TV, that tends to overstimulate kids and make them more challenging). If he has problems being told 'no,' just find a way to do it less. (no, I'm not insane!) Set up your home and thus life so that he doesn't have to hear 'no.' Remove items from his vision that he can not have. If he proposes something (say running in the parking lot) that is a total 'no' then just bend down and give him an option (which would you like? I can hold you or you can ride in the cart) He is old enough to understand.
If worse comes to worse...l just ignore it (to an extent, frequently check on him, it only takes a minute for trouble to happen) and above all, just love him and know that this stage will pass. Helping him feel loved, secure and stimulated will help that happen faster.
My best wishes to you and your family.