There is a big difference between a unisex swimming pool changing room that has individual cubicles and women-only spaces where people get undressed in a communal area. As sixthswan says, unisex facilities probably shouldn't exist but at least they still provide a degree of privacy because people get undressed in cubicles.
That's not the same as spaces where women get undressed or bathe naked in front of each other. At the Wi Spa incident in LA last year, several women complained to management that a trans-identifying male (TIM) in the changing room was deliberately exposing his penis in an area where women bathed naked. The immediate reaction on social media was one of sympathy for the TIM - the women who complained were dismissed as bigots and transphobes. It later transpired that that the TIM had a history of sex offences. He was charged with five counts of indecent exposure relating to the incident.
There is also a growing problem of TIMs being housed in women's prisons and, once there, committing sexual assaults. For example, Karen White (aka Stephen Wood) sexually assaulted two inmates at a women's jail. He had been sent there despite having previous convictions for sexual offences against women (
https://www.bbc.co.uk/news/uk-england-leeds-45825838).
Some women may be ok with having men in spaces where they are vulnerable, but there are many who are definitely not ok with it. It's about privacy as well as safety - many just don't want to be looked at by men when they are naked because it makes them feel vulnerable. I'm on their side, not on the side of men who want to invade their spaces.