My current youngest son (my wife is expecting in late August) is about the same age as your son. He will be 4 in September. I think it really differs by kid and you kind of have to read their body language to tell. My oldest was hitting with an aluminum bat in the batting cages by age 4. My soon to be 4 year old still enjoys hitting with the Fred Flintstone bat and a tennis ball. If you're letting him hit off of a tee, I'd think you could have him use a slightly more normal bat. They make real light t-ball bats now, some as light as 13 oz. That might be a good next step for him. If you are going to soft toss to him, I'd let him keep the bat he's using, but move back further when you pitch. That's the plan I've used with my youngest and he's still hitting pretty well and I've backed up to about 3 steps shy of where they throw for coach pitch. I've moved back slowly week by week, though and had some weeks where he struggled and I didn't move back at all until he "dominated" that distance.
Also to add, if you do buy one of the aluminum t-ball bats and you notice that his swing is completely screwed up and his mechanics are awful because he's using every ounce of strength to get the bat around, switch back to the lighter whiffle ball bat. Kids at 3-4 years old aren't the strongest. Better to have him keep his good mechanics with a whiffle ball bat then screw his mechanics totally up by trying to swing a bat that is too heavy for him.
The most important thing is that he enjoys himself, though. If by his body language he's telling you he's not having fun, quit and try another day. It's such a wonderful thing to play catch with your kids, though. I know. I'm addicted.