Not height so much, but if a guy is only 130 lbs, even at age 17, there is only so much room for him to fill out.
Tools like hitting and fielding are important, but how do you measure them? Only through stats? If that were the case there'd be no need for scouts, you'd just promote the players who have performed the best.
The reason every one in the minor league system projected Drew Stubbs as a potential 20 HR guy, even though he couldn't hit 20 homers at various stages in the minors if you spotted him the first 15 is because of his physical tools.
Torreyes may be the exception, but the exception gets treated differently than the rule. Whereas the rule is treated as a top propsect until he proves that he isn't, the exception won't be treated as a top prospect until he proves he is. And even then there will be skepticism.