This is exactly how it should be.
There is a lot to be said for watching your own children. I hate the people who are overly sensitive about safety. For your children, I understand worrying constantly even though I have none. I can't imagine what it would be like to constantly worry about their safety, but there is a point at which you have to understand you are being overly protective. How many children are kidnapped or lured by kidnappers when their parents are paying any attention at all. I would say close to none.
So, I'm not allowed to enjoy a day at a park because I don't have any children? Thats just as discrimitive as thinking all people of middle eastern appearance are terrorists.
I'm not saying you shouldn't keep an eye out for things that might be indicators of something criminal. I'm just saying that when you get that bad feeling, you avoid any contact with that person, and you have the responsibility of getting you kids away from what you feel as a threat, not taking it upon yourself to decide who is a bad person and immediately assume you are in the right. Those are the people who drive me nuts. The ones who assume they are right, and if they aren't you shouldn't have been there anyway. Luckily, there are few people out there who act this irrational, but everyone knows somebody like that.