There's a huge difference between 'You're so unattractive I physically recoil at the concept of touching you' and 'you're physically fine but a great person, so I'll look past averageness'. Provided you aren't in the first group, women won't usually care. As to how shallow men are, I'm unsure.