Bah! I flit between man, woman, agender, and pissoff with regularity, and my body shape, tone of voice, chromosomes, genitals, or anything else have nothing to do with it.
Although if I lose a lot of weight I'm going to look into HRT. But one thing at a time, and the weight is more important.
So screw them, our thought process is yours, and not theirs. Human gender (and sexuality) is not as solid and unchanging as people think. If you ever stop thinking of yourself as a man and start thinking of yourself as a woman then congratulations, you are now a woman, whether or not anything has changed