Yeah, I agree about the sample size. The guy on the frowning side of the D with a large enough sample size is Bronson. But, his FIP is almost the same as his ERA so I don't think he's being hurt too badly by the D.
That does go to your point because when he's off, aside from Owing, I think he's the starter that gets hit the hardest.