It was your decision to take the job and you represent the company you work for. If you don't like the job, get another one. If you can't find another one, go to trade school. If you don't have time for trade school because of family or something, well, you probably had opportunity when you were younger and didn't take advantage of it.
It ain't the guy on the other end of the phone's fault. You're the one that's stalking him, so he's got every right to be pissed.