Some evidence to back this up.
Divide people up by gender and sexual preference. You have four groups. Straight Men, Straight Women, Gay Men and Gay Women.
Of those four groups, which ones spend money and time making themselves look attractive for potential mates? Who works out, stays fit, dresses well, spends money and time on their hair and other physical attributes? Straight Women and Gay Men. Who are they trying to attract? Men.
Which groups don't spend much time on their appearance? Straight Men and Gay Women. Who are they trying to attract? Women.
This suggests that there is something biological in men that makes them most concerned with physical attraction, and something biological that makes women most concerned with personality, character, wealth, etc.