"...a step towards liberalization and secularization" [in Israel]?
In fact, there are a variety of disorders were men don't have XY, and women don't have XX, despite being identified at birth as male or female respectively. There are combinations that aren't XX or XY. There are also cases of people born with ambiguous genitalia.
Penis/XY are partial markers, indicators, of the sex and gender. They do not determine it in all cases. There are other physical markers that serve as indicators but which may not determine the sex/gender on their own either. Hormone levels. Brain structure. Reproductive system. Secondary sex characteristics. Genitals and chromosomes are part of an array. There's been evidence that some transgendered individuals have brain structures opposite their "birth sex" (as identified by genitalia). So does that mean it's a physical or "mental" condition?
In fact, can anything be purely mental? Or is every thought and instinct a product of a physical process? Well, of course it's physical. Neurons fire, neurotransmitters flow, etc. Physical.
Furthermore, there are social indicators that determine whether someone is a "man" or "woman" regardless of whether they are "male" or "female". The way someone dresses is an indicator. The way someone was raised. The way someone communicates. These factors are sometimes at odds with one another. They are sometimes at odds with traditional physical markers. That's why it ultimately comes down to the way someone identifies.
So a baby born with a penis and XY will appear to be male, and will be raised as male. At puberty, perhaps testosterone starts surging and adds further male characteristics such as beard growth, etc. The person now has a male upbringing, male genitals, male hormones, male reproductive system. But what if their brain is female? What if then this person begins gender transition and takes female hormones and medicine to stop testosterone production? Then they'll be hormonally: female, brain structure: female, genitally: male, etc... Then they change their genitals, and remove their reproductive system... at some point they have a different composition of male and female markers than they started with. When is it enough to tilt the balance from "male" to "female"? And is it directly tied to whether they identify as "man" or "woman"? And is it tied to whether others identify them as such? These are rhetorical questions meant for you to see that the link between our social constructs and are biologies are not always straightforward, and that even "just" our biologies are an amalgam of many many factors. This is the 21st century, but the average understanding of gender and sex is stuck in the 20th, or earlier.
Wikipedia has a good rundown on gender identity and you'll find further explanations and answers to many of your questions.